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I. INTRODUCTION 

The American Recovery and Reinvestment Act of 2009 (ARRA) provided an unprecedented 
amount of federal funds for education in an effort to lessen the effects of the nation’s economic 
downturn and to make a lasting investment in education. Through $97.4 billion in ARRA funds, 
the federal government sought to save education jobs, fund a new wave of innovation in 
education, and support comprehensive efforts to turn around low-perfonning schools. The 
School Improvement Grants (SIG) program received an additional $3 billion through ARRA. 
Through fonnula-based grants to states, SIG focuses on turning around low-perfonning schools 
(formally refened to as “persistently lowest-achieving schools” in SIG guidance) using one of 
four school intervention models. The SIG application criteria laid out school improvement 
practices in four main areas: (1) adopting comprehensive instructional refonn strategies, 
(2) developing and increasing teacher and principal effectiveness, (3) increasing learning time 
and creating community-oriented schools, and (4) having operational flexibility and receiving 
support. 

To leam about the effectiveness of SIG, the U.S. Department of Education’s (ED’s) Institute 
of Education Sciences commissioned an evaluation of the program. The SIG evaluation is based 
on a descriptive analysis of school-level education practices and a regression discontinuity 
design to assess the effect of SIG on student outcomes (the latter of which will be presented in a 
future report). 

At the request of ED’s Office of English Language Acquisition (OELA), part of the 
evaluation also focuses on how districts and schools have addressed the needs of English 
language learners (ELLs) as they used the practices promoted by SIG. ELLs are of particular 
interest to this evaluation because: (1) they are historically lower-achieving than non-ELLs 1 , and 
(2) the SIG program placed particular emphasis on prioritizing the academic achievement of 
high-needs students, including ELLs (U.S. Department of Education 2010). 

The study is being conducted by Mathematica Policy Research, the American Institutes for 
Research, and Social Policy Research Associates. In this chapter we provide background 
infonnation about the SIG program, present prior research on SIG, and provide an overview of 
our evaluation and the contents of this report. 

A. Scope, Purpose, Timing, and Size of SIG Funding 

The SIG program aims to support the implementation of school intervention models in low- 
perfonning schools. This evaluation focuses on SIG awards granted in 2010, when roughly 
$3.5 billion in SIG awards were made, with $3.0 billion of that funding from ARRA. To receive 
SIG in 2010, state education agencies (SEAs) had to submit applications to ED identifying SIG- 
eligible schools (based on criteria specified by ED) and specifying the criteria the SEA would 
use to make subgrants of SIG to eligible districts. SIG funds were awarded in grants to states 
(apportioned by a formula based on Title I allocations). States were then required to distribute 95 
percent of those funds through competitive subgrants to local education agencies (LEAs, which 


1 Since 2002, ELLs’ reading test scores have been below those of non-ELLs on the National Assessment of 
Educational Progress test (National Center for Education Statistics. The Condition of Education. Accessed February 
17, 2014 at https://nces.ed.gov/programs/coe/indicator_cgf.asp). 
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typically are school districts) for implementation of school intervention models in eligible 
schools over the course of three school years starting with 2010-2011. 

For the 2010 SIG competition, ED required states to categorize eligible schools into three 
eligibility tiers based on each school’s level (elementary or secondary), eligibility for and receipt 
of Title I program funds, 2 3 4 and achievement or graduation rate. These tiers helped prioritize the 
distribution of SIG funds at the local level and detennined the strategies to be used for school 
turnaround. Generally, schools eligible for SIG under Tier I and Tier II: 

1. Were receiving or eligible for Title I 

2. Were in improvement, corrective action, or restructuring under No Child Left Behind 
(NCLB) and 

a. Were among the lowest achieving 5 percent of schools (or the lowest achieving 
5 schools, whichever is greater) in the state, based on proficiency rates in 
English/language arts (ELA) and mathematics, or 

b. Were high schools with a graduation rate under 60 percent; and 

3. Had failed to show progress over a number of years. 3,4 

Tier III schools were the remaining Title I-receiving schools in improvement, corrective action, 
or restructuring under NCLB, and the remaining Title I-eligible schools, that did not meet Tier I 
or Tier II requirements. Tier I and II schools had to be prioritized over Tier III schools for 
awards. 

ED required that each SIG-awarded school under Tier I or Tier II implement one of four 
school intervention models, each of which featured specific practices: 

• Transformation. This model requires schools to replace the principal, adopt a 
teacher and principal evaluation system that accounts for student achievement growth 
as a significant factor, adopt a new governance structure, institute comprehensive 
instructional reforms, increase learning time, create a community-oriented school, and 
have operational flexibility. 

• Turnaround. This model requires schools to replace the principal, replace at least 50 
percent of the school staff, institute comprehensive instructional refonns, increase 
learning time, create community-oriented schools, and have operational flexibility. 

• Restart. This model requires schools to convert to a charter school or close and 
reopen under the management of a charter management organization or education 
management organization. 


2 Title I, Part A (Title I) of the Elementary and Secondary Education Act provides financial assistance to LEAs 
and schools with high numbers or high percentages of children from low-income families. Title I funds are allocated 
using formulas that are based primarily on census poverty estimates and the cost of education in each state. 

3 The number of years over which progress was to be assessed was left to the discretion of SEAs but had to be 
at least two years. 

4 The main difference between Tier 1 and Tier II was that Tier I generally included Title I-receiving elementary 
or secondary schools identified for improvement, whereas Tier II generally included secondary schools that were 
eligible for, but did not receive, Title 1 funds. 
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• School closure. This model requires schools to close and for their students to enroll 
in higher-achieving schools in the district. 

The distribution of the 2010 SIG grantees from Tiers I, II, and III across model types and the 
distribution of award amounts are shown in Table 1.1. The maximum grant amount was 
$2 million per year for three years (or $6 million in total over three years). The most commonly 
selected school intervention model was the transfonnation model (623 of 1,253 schools, 
50 percent) with a median award per school of $2.1 million over three years. The turnaround 
model was the second most popular school intervention model (178 of 1,253 schools, 14 percent) 
with a median award of $2.7 million. The restart and closure models were selected for just 
33 and 16 schools (3 percent and 1 percent). A total of 403 Tier III schools received awards to 
implement Tier III strategies, and the median award among those schools was $300,000 
(Hurlburt et al. 201 1). 5 


Table 1.1. SIG Funding Awarded in 2010 and Number of Schools Implementing Each Intervention Model 




School Intervention Model 




Transformation 

Turnaround 

Restart 

Closure 

Tier III 
Strategies 3 

Number of Schools Implementing Each Intervention Model 

Tier 1 

354 

138 

24 

8 

0 

Tier II 

255 

40 

9 

8 

0 

Tier III 

14 

0 

0 

0 

403 

Total 

623 

178 

33 

16 

403 

Distribution of Award Amounts (Over Three Years) 

10th Percentile 

$942,892 

$1,236,632 

$1,187,500 

$31,935 

$60,190 

50th Percentile 

$2,100,000 

$2,684,490 

$2,167,965 

$50,000 

$300,000 

90th Percentile 

$5,114,190 

$5,190,000 

$5,490,491 

$254,323 

$900,405 


Source: IES database of SIG grantees; Hurlburt et al. (201 1 ). 

Note: The SIG awards summarized in this table are from the round of state applications due to the U.S. 

Department of Education on February 8, 2010. The award amount percentiles are based on the total 
award amount per school. 

a Tier III strategies refer to all school improvement strategies used by SIG-awarded Tier III schools. Federal rules did 
not require Tier III schools to implement one of the four ED-specified school intervention models. 

B. Prior Research on SIG 

Prior research on SIG implementation is limited. Several studies investigated states’ 
methods for selecting SIG schools and their plans for monitoring and supporting them (Center on 
Education Policy 2011a, 2011b, 2012a, 2012b; Government Accountability Office 2011; 
Hurlburt et al. 2011, 2012; U.S. Department of Education 2012a). States reported providing 
schools and districts with various types of assistance to aid with SIG implementation, but many 
states also planned to provide some assistance to low-perfonning, non-SIG schools (Center on 
Education Policy 2011a, 2012b). One study that compared school improvement practices used by 
SIG and non-SIG schools found that both groups reported implementing similar practices, 


5 Tier III strategies refer to all school improvement strategies used by SIG-awarded Tier III schools. Federal 
rules did not require Tier III schools to implement one of the four ED-specified school intervention models. 
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though SIG schools reported implementing these practices more intensively or supplementing 
them (Center on Education Policy 2012a); another similar study found that, for 16 of 32 SIG- 
promoted practices examined, schools implementing a SIG model were statistically significantly 
more likely than schools not implementing one to report using that practice (Hernnann et al. 
2014). A case study of 25 SIG schools found that approaches to leadership varied across schools, 
with most principals exhibiting a mix of leadership qualities (Le Floch et al. 2014). Common 
challenges encountered during SIG implementation include difficulties attracting and retaining 
high-quality teachers and principals, particularly in rural areas; difficulties using data to inform 
and differentiate instruction; and limited state capacity to provide assistance (Center on 
Education Policy 2012a; GAO 2011; U.S. Department of Education 201 la-j). 

This report focuses on the usage of SIG-promoted policies and practices, and sets the table 
for a future report that will examine whether receipt of SIG funds has an impact on student 
outcomes. Because this report does not examine the effectiveness of SIG, it is beyond the scope 
of this report to discuss the literature on SIG effectiveness in detail; however, we provide a brief 
summary of it in the next two paragraphs to set the stage for the future report. 

Prior research on the effectiveness of the SIG program is limited, but the available research 
generally finds a positive relationship between SIG and student achievement. A study from 
California found a positive relationship between SIG and a school-level academic perfonnance 
index that was based on student tests in English, math, social studies, and science; these positive 
results were particularly true for schools that implemented the turnaround model (Dee 2012). 
Similarly, a study from Philadelphia found that the SIG turnaround and restart models were 
positively associated with increased reading and math test scores in elementary and middle 
schools, though the turnaround model had no association with test scores for high schools 6 (Gold 
et al. 2012). A descriptive analysis showed that after the first year of implementation, two-thirds 
of SIG schools nationally experienced gains in math and another two-thirds experienced gains in 
reading (U.S. Department of Education 2012b). Another descriptive analysis found that gaps in 
the percentages of students scoring at or above proficient on state assessments between SIG- 
awarded schools and two comparison groups (SIG-eligible schools that did not receive grants 
and non-SIG-eligible schools) narrowed over the first two years of the grants (Council of the 
Great City Schools 2015). 

Prior studies on the types of practices promoted by SIG provide no conclusive evidence on 
whether these practices improve student outcomes. In all four SIG areas (using comprehensive 
instructional refonn strategies, developing and increasing teacher and principal effectiveness, 
increasing learning time and creating community-oriented schools, having operational flexibility 
and receiving support), both experimental and non-experimental studies found mixed results. 7 
Some studies found that the types of practices promoted by SIG in those areas were associated 
with improved student outcomes, while other studies found no relationship between these 


6 The turnaround model was implemented in only two of the high schools evaluated in this study, however. 

7 Experimental studies are those that examine treatment and comparison groups from a randomized controlled 
trial. The results from experimental studies (if such studies are conducted appropriately) can be used to make causal 
statements about the effect of a practice, policy, program, or intervention on an outcome of interest. The results from 
nonexperimental studies cannot be used to make such causal statements because these studies cannot rule out the 
possibility that other differences between the groups — besides the intervention itself — caused any observed 
differences in outcomes. 


4 



I. Introduction 


Usage of Practices Promoted by SIG 


practices and student outcomes. 8 Studies on the effectiveness of school turnaround more broadly 
(excluding SIG intervention models, but including school intervention models similar to those 
promoted by SIG) also found mixed results. 9 

C. Evaluation Focus 

As noted in section B above, there is no conclusive evidence on whether the types of 
practices promoted by SIG improve student outcomes, and few studies on the implementation of 
SIG-promoted practices examine whether the practices used by SIG schools differ from those 
used by schools that did not receive SIG. This evaluation seeks to address these gaps in the 
existing literature. It is designed to examine whether the program affects student achievement (a 
topic that will be addressed in a future report), whether SIG recipients implementing school 
intervention models report using the practices promoted by the program, and whether they are 
using these practices to a different extent than schools that have not received such grants to 
implement school intervention models. This volume of the report focuses on the following 
research questions: 

• Are SIG-funded intervention model schools using the improvement practices 
promoted by the four SIG intervention models, and how does that compare to the 
usage of those practices by schools not implementing a SIG-funded intervention 
model? 

• Does usage of these practices include a focus on ELLs and does that focus on ELLs 
vary between schools implementing a SIG-funded intervention model and schools not 
implementing one? 

• Does usage of these ELL-focused improvement practices vary by the percentage of 
students who are ELLs or the ELL/non-ELL student achievement gap? 

To address these research questions, we analyzed data from surveys of school administrators 
from approximately 470 schools. 10 The sample of schools was purposively selected to support 
the estimation of impacts of SIG-funded models on student outcomes (which will be presented in 
a future report). Though the results from this evaluation of SIG are not necessarily generalizable 
to SIG schools nationwide, they are nonetheless important because they add to the limited 
knowledge base about the implementation and impacts of SIG-funded school turnaround efforts. 

D. Looking Ahead 

In Chapter II, we describe the study sample, design, and data collected to address these 
research questions. In Chapter III, we provide baseline information on the SIG sample. In 


8 To cite a few examples: Abdulkadiroglu et al. 2011; Allen et al. 2011; Angrist et al. 2011; Betts et al. 2005; 
Black et al. 2009; Carlson et al. 2011; Clark 2009; Clark et al. 2013; Constantine et al. 2009; Cortes et al. 2012; 
Decker et al. 2004; Dobbie and Fryer 2011; Furgeson et al. 2012; Garet et al. 2010; Glazerman et al. 2006; Gleason 
et al. 2010; Flenderson et al. 2007; Henderson et al. 2008; James-Burdumy et al. 2005; May and Robinson 2007; 
Quint et al. 2008; Slavin et al. 2011; and Steinberg 2014. 

9 To cite a few examples: Bifulco et al. 2003; Borman et al. 2003; Booker et al. 2009; de la Torre and Gwynne 
2009; Dobbie and Fryer 2011; Fryer 2014; Hoxby et al. 2009; Hoxby and Rockoff2005; Gleason et al. 2010; Player 
and Katz 2013; Strunk et al. 2012; Tuttle et al. 2013; Zimmer and Buddin 2006; and Zimmer et al. 2012. 

10 Following reporting requirements established by the U.S. Department of Education’s National Center for 
Education Statistics, we rounded all sample sizes to the nearest ten. 
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Chapter IV, we present findings on the extent to which SIG-funded model schools reported using 
the improvement practices promoted by SIG in spring 2012 and how that compares to schools 
not implementing a SIG-funded model. In Chapter V, we present findings on the extent to which 
schools implementing a SIG-funded model and those not implementing one reported focusing on 
ELLs in their usage of the practices promoted by SIG, and how that varies by the percentage of 
students who are ELLs and the ELL/non-ELL achievement gap. In Chapter VI, we discuss the 
findings from this volume. In Appendices A through E, we provide additional results, including 
details on responses to individual survey questions. 


6 



II. Study Sample, Data Collection, and Analysis 


Usage of Practices Promoted by SIG 


II. STUDY SAMPLE, DATA COLLECTION, AND ANALYSIS 

In this chapter, we describe the study sample, the data collected, and the method of 
analyzing the data for the evaluation of SIG. 

A. Study Sample 

The sample for the evaluation of SIG includes 470 low-performing schools within 60 
districts across 22 states. 11 These schools were purposively selected to support the estimation of 
impacts of SIG-funded models on student outcomes (which will be presented in a future report); 
that is, the SIG sample was not randomly selected. Thus, caution should be taken when 
interpreting the results. In particular, it should not be assumed that the findings presented in this 
volume of the report necessarily generalize to schools nationwide. 

The sample includes more than a third of all schools implementing a SIG-funded model in 
2011-2012. Low-performing schools (formally referred to as “persistently lowest-achieving 
schools” in SIG guidance) are generally schools that (1) are either Title I-receiving schools 
identified for improvement or Title I-eligible schools, and (2) fall in the lowest 5 percent in 
academic achievement in the state (or, for high schools, that have a graduation rate under 60 
percent) for a number of years. 12 We present more infonnation on the characteristics of the 
selected schools in Chapter III. 

B. Data Collection 

The data examined in this volume come from surveys of school administrators conducted in 
spring 2012. 13 We conducted a web survey of school administrators at the 470 schools in the SIG 
sample. The survey was sent to the principal of each school, with instructions indicating that he 
or she should consult with other school staff before completing the survey if needed, resulting in 
one response per school. The survey collected information about the SIG models and specific 
practices reported by schools as well as supports they reported receiving from states and districts. 
The SIG objectives in each area and the practices within each area for which we had survey data 
are detailed in Table II.l. 14,15 The school survey included questions addressing six ELL-focused 


11 The school sample size for this report (470 schools) is slightly larger than the sample of 450 schools used in 
Herman et al. (2014) because we were able to determine the status (whether or not they implemented a SIG-funded 
model) of roughly 20 additional schools, after the analyses for Herman et al. were completed but before the analyses 
for this report had begun. 

12 The number of years over which progress was to be assessed was left to the discretion of SEAs but had to be 
at least two years. 

13 The study also included interviews with state and district administrators. To provide context for the analysis 
in Chapter IV about schools ’ reported adoption of practices promoted by SIG, in Appendix B of this volume, we use 
the district interview data to summarize the extent to which districts reported adopting the practices promoted by 
SIG. In Appendix C, we present detailed responses to individual school survey questions about the adoption of 
practices promoted by SIG and to individual state and district interview questions about the supports provided to 
schools by states and districts. 

14 The school administrator survey protocol is available at http://www.mathematica- 
mpr.com/publications/PDFs/Spring_20 12_School_Administrator_Survey.pdf. 

15 The survey questions specified that the 201 1-2012 school year was the time period of interest. Therefore, the 
results in this report represent a point-in-time measure of the practices used by schools. Throughout the report, we 
use phrases such as “In spring 2012, schools reported using practices” to reflect that these are point-in-time 
measures. Because the usage of practices is a dynamic process, it is possible that some practices used by schools at 
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practices aligned with SIG objectives (Table II. 2). To limit the length of the survey, we sought 
input from IES and the SIG program office on which questions were of greatest interest. We 
conducted pilot tests of the instrument and then revised the survey questions as needed to ensure 
the unifonnity and consistency of the data collected. Eighty-seven percent of schools in the 
sample responded to the survey. 

Table 11.1. SIG Objectives and Practices Addressed by School Administrator Survey Questions, by 
Topic Area 


Topic Area 


SIG Objectives Practices Addressed by School Administrator Survey Questions 


Implementing 
comprehensive 
instructional reform 
strategies 


Developing and 
increasing teacher 
and principal 
effectiveness 


Using data to identify and 
implement an instructional 
program 

Promoting the continuous use 
of student data 

Conducting periodic reviews to 
ensure that the curriculum is 
being implemented with fidelity 

Implementing a new school 
model (such as a themed 
academy) 

Providing supports and 
professional development to 
staff to assist ELLs and 
students with disabilities 

Using and integrating 
technology-based supports 

Tailoring strategies for 
secondary schools 


Using rigorous, transparent, 
and equitable evaluation 
systems 

Identifying and rewarding 
effective teachers and 
principals and removing 
ineffective ones 

Providing high-quality, job- 
embedded professional 
development or supports 


Implementing strategies to 
recruit, place, and retain staff 


• Use data to evaluate instructional programs 

• Use data to inform instruction 

• Use benchmark or interim assessments at least annually 
No items in school survey aligned with this objective 3 

No items in school survey aligned with this objective 3 


• Implement strategies for English language learners (ELLs) to 
master content 


• Increase technology access or use computer-assisted instruction 

• Track postsecondary preparation or use project-based learning 

• Create small learning communities or academies 

■ Track progress to high school graduation 

• Use student achievement growth 

• Use multiple evaluation measures 

• Use evaluations to inform compensation 

• Review competencies of staff or replace instructional staff 


• Provide multiple-session professional development events b 

• Provide professional development on Common Core State 
Standards (CCSS), state standards, or turnaround 

• Professional development involves working collaboratively or is 
facilitated by school leaders 

• Provide professional development on student learning needs 

• Design professional development with school staff 5 

» Use data to evaluate professional development 

• Provide financial incentives or flexible work conditions 

• Use evaluation results to inform reductions in force or have policies 

that allow principal authority to hire staff b 


(continued) 

the time of the survey were no longer in use after the survey. For example, some schools that used the Common 
Core standards at the time of the survey may reside in states that repealed those standards after the survey. 
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Topic Area 


SIG Objectives Practices Addressed by School Administrator Survey Questions 


Increasing learning Increasing learning time 

time and creating 

community-oriented Engaging families and 

schools communities and providing a 

safe school environment that 
meets students’ social, 
emotional, and health needs 


• Use schedules or strategies to increase learning time 

• Change parent or community involvement strategies 

• Provide professional development on working with parents or 
cultural sensitivity, or increase volunteers or safety measures 

• Change discipline policies 

■ Use data to guide nonacademic supports 


Having operational flexibility • Have autonomy on budgeting, hiring, discipline, or school year 

length 

Receiving technical assistance . Receive training or technical assistance to support school 
and support improvement or use data to improve instruction 

Source: SIG application; surveys of school administrators in spring 2012. 

a The number of questions included in the school administrator survey was purposefully limited to reduce the time it took to complete 
the survey. We initially developed the interview questions based on an examination of the SIG application criteria. To ensure that 
the interview was of a reasonable length, we then pared down the initial list of questions through a deliberative process with the 
Institute of Education Sciences and the SIG Program Office, to assess their priorities for the types of questions to include. The 
survey did not include any questions about this objective. 

b The school administrator survey did not ask about this practice for principals. 


Having operational 
flexibility and 
receiving support 


Table 11.2. SIG Objectives and the ELL-Focused Practices Aligned with Those Objectives That 
Were Addressed by School Administrator Survey Questions, by Topic Area 


Topic Area 


ELL-Focused Practices Addressed by School Administrator 
SIG Objectives Survey Questions 


Implementing 
comprehensive 
instructional reform 
strategies 


Developing and 
increasing teacher 
and principal 
effectiveness 


Using data to identify and 
implement an instructional 
program 

Promoting the continuous use 
of student data 

Conducting periodic reviews to 
ensure that the curriculum is 
being implemented with fidelity 

Implementing a new school 
model (such as a themed 
academy) 

Providing supports and 
professional development to 
staff to assist ELLs and 
students with disabilities 

Using and integrating 
technology-based supports 

Tailoring strategies for 
secondary schools 

Using rigorous, transparent, No items in school survey aligned with this objective for ELLs a 

and equitable evaluation 

systems 

Identifying and rewarding No items in school survey aligned with this objective for ELLs a 

effective teachers and 
principals and removing 
ineffective ones 

Providing high-quality, job- No items in school survey aligned with this objective for ELLs a 

embedded professional 
development or supports 

Implementing strategies to • Offer financial incentives for teachers with ELL expertise 

recruit, place, and retain staff « Offer financial incentives for principals with ELL expertise 


No items in school survey aligned with this objective for ELLs a 

• Use data on ELLs to inform instruction 

No items in school survey aligned with this objective for ELLs a 

No items in school survey aligned with this objective for ELLs 3 

• Implement strategies for ELLs to master content 

No items in school survey aligned with this objective for ELLs 3 
No items in school survey aligned with this objective for ELLs a 
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Topic Area 

SIG Objectives 

ELL-Focused Practices Addressed by School Administrator 
Survey Questions 

Increasing learning 
time and creating 
community-oriented 
schools 

Increasing learning time 

Engaging families and 
communities and providing a 
safe school environment that 
meets students’ social, 
emotional, and health needs 

No items in school survey aligned with this objective for ELLs a 
• Provide additional services for ELLs 

Having operational 
flexibility and 
receiving support 

Having operational flexibility 

Receiving technical assistance 
and support 

No items in school survey aligned with this objective for ELLs a 
• Receive supports to use data on ELLs to improve instruction 


Source: SIG application; surveys of school administrators in spring 2012. 


ELL = English language learner. 

a The number of questions included in the school administrator survey was purposefully limited to reduce the time it took to complete 
the survey. We initially developed the interview questions based on an examination of the SIG application criteria. To ensure that 
the interview was of a reasonable length, we then pared down the initial list of questions through a deliberative process with the 
Institute of Education Sciences and the SIG Program Office, to assess their priorities for the types of questions to include. The 
survey did not include any questions about this objective. 


To provide baseline and other contextual information for the sample used in this report, we 
drew on publicly available data from the Common Core of Data, which includes annual data 
about each public school, LEA, and state in the country. We accessed the Common Core to 
obtain 2009-2010 school-level characteristics such as the percentages of students in each 
race/ethnicity category, of students eligible for free or reduced-price lunch, of schools eligible 
for Title I, and of schools at each level (elementary, middle, high). 

C. Analysis Methods 

In this section, we describe the methods we used to compare the practices reported by 
schools implementing a SIG-funded model and schools not implementing one. The purpose of 
these comparisons is not to detennine whether receipt of SIG to implement a school intervention 
model caused schools to use particular practices, but simply to determine whether schools 
implementing a SIG-funded model used the practices promoted by the four SIG models and how 
their experience compares with the usage of those practices by schools not implementing a SIG- 
funded model. In interpreting the results, please note the following caveats: (1) the findings are 
based on self-reported usage of practices, (2) our study instruments did not address every 
practice listed in the SIG application, (3) the application wording left it up to the schools to 
decide many of the details about how to implement particular practices, (4) we did not collect 
information about the quality or fidelity with which the practices were implemented, and (5) the 
SIG sample was not randomly selected. 16 For all of these reasons, the SIG findings should be 
interpreted with caution. 


16 Here we provide several potential explanations for how these limitations might affect the data. Our data do 
not allow us to determine whether any of these possible explanations are correct, but we offer them as starting points 
for thinking about how the results might be affected by the limitations. Self-reported levels of practice usage might 
be overestimated (relative to actual usage) if schools provided socially desirable responses. This would likely lead to 
overestimated levels for all schools. The study team took several steps to ensure that schools provided accurate 
responses, including telling schools that the survey was not an audit and that we would report aggregated responses 
across schools, rather than singling out any individual school. The fact that self-reported levels of practice usage (as 
presented in Chapter IV) are not all 100 percent, and in many cases are much lower than 100 percent, suggests that 
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The practices that we examined were either required or permissible under the SIG 
transfonnation and turnaround models. Schools that implemented the restart model under SIG 
were required to convert or close and reopen under a charter school operator, a charter 
management organization, or an education management organization that was selected through a 
rigorous review process. However, these schools could still choose to use the practices that were 
either required or permissible by SIG under the transformation and turnaround models, so we 
include them in the analysis. Only 20 of the 470 schools in the sample implemented the restart or 
closure model, so it is unlikely that their inclusion or exclusion would have a substantial effect 
on the overall results. 

We first describe how we fonned the two groups that are the basis for the comparisons 
presented in this volume of the report. We then describe how we summarized the large number 
of findings and how we analyzed the extent to which schools focused on ELLs in their usage of 
practices promoted by SIG. 

1. SIG Comparisons 

We divided the SIG sample of 470 low-perfonning schools into two groups for the purposes 
of this report. The first group includes roughly 290 schools that indicated they received SIG 
funding and were implementing one of the four school intervention models in spring 2012. We 
refer to these schools as schools implementing a SIG-funded intervention model in 2011-2012 
(or schools implementing a SIG-funded model, for short). The second group includes the 
remaining 180 schools in our sample, which includes both those that did not receive SIG funding 
and those that received SIG funding but were not implementing one of the four intervention 
models. We refer to this second group of schools as schools not implementing a SIG-funded 
intervention model in 2011-2012 (or schools not implementing a SIG-funded model, for short). 
The analyses presented in this volume of the report focus on comparisons of these two groups. 17 


(continued) 

the issue of providing socially desirable responses is not a rampant problem in our data. However, it is possible that 
schools implementing a SIG-funded model might have been more likely than schools not implementing one to 
provide socially desirable responses, given that they received SIG funds to implement the practices we examined. 
Therefore, the results for schools implementing a SIG-funded model might be more inflated than the results for 
schools not implementing one, so readers should use caution when interpreting the results. Regarding the fact that 
our study instruments did not address every practice listed in the SIG application criteria, this is unlikely to have a 
large effect on the overall results because there were very few practices not addressed by our survey questions 
(Table II. 1 shows that only 2 out of 15 SIG objectives were not addressed by survey questions). Regarding the 
quality or fidelity with which the practices were implemented, and the fact that the application wording left it up to 
the schools to decide many of the details about how to implement particular practices, our data might overestimate 
usage levels if schools tended to report that they used a practice if they had at least begun to use it, but had not 
necessarily implemented it fully; this would lead to overestimated levels for all schools, rather than affecting the 
differences between schools implementing a SIG-funded intervention model and schools not implementing one. In 
addition, as noted above, the fact that many of the self-reported levels of practice usage (as presented in Chapter IV) 
are well below 100 percent suggests that these levels might not be overestimated. 

17 For an examination of the implementation of SIG conducted separately for schools that implemented the 
transformation and turnaround models, please see Herrmann, M., L. Dragoset, and S. James-Burdumy. “Are Low- 
Performing Schools Adopting Practices Promoted by School Improvement Grants?” NCEE 2015-4001. Washington, 
DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and 
Regional Assistance, October 2014. 
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The construction of these two groups of schools was driven by the fact that this evaluation is 
focused on SIG-funded intervention models. More specifically, because this evaluation focuses 
on Tier I and II SIG schools in particular, and the SIG “intervention” for those schools consisted 
of using SIG funds to implement one of four ED-specified school intervention models, the 
analysis focuses on comparing schools that implemented a SIG-funded intervention model to 
schools that did not. We placed the 10 schools that received SIG funding but were not 
implementing a SIG model into the second group (that is, the group of schools not implementing 
a SIG-funded model) because they would not be expected to have used the practices promoted by 
the four SIG models. We also placed into this second group the 10 schools that reported 
implementing a SIG model without SIG funding because the goal of this report is to compare the 
practices used by schools implementing SIG-funded models with practices used by similar 
schools that were not implementing these SIG-funded models. 18 

2. Summarizing Findings from the School Survey 

Given the large number of questions in the survey, it is difficult to discern broad patterns or 
form overall conclusions by only separately examining responses to individual questions. 
Therefore, we analyzed data from the survey using methods designed to provide infonnation 
about broad patterns observed in the data. Readers interested in the responses to specific survey 
questions can refer to Appendix C. The process of summarizing findings involved several steps: 

1. Selecting subtopics. For each of the four broad topic areas in the SIG application 
criteria (listed in Chapter I and Table II. 1 above), we selected subtopics of interest 
using the SIG application criteria as a guide. For example, the section of the 
application criteria focusing on comprehensive instructional reform strategies 
identified activities in seven subsections: (1) using data to identify and implement an 
instructional program, (2) promoting the continuous use of data to identify and 
address the needs of individual students, (3) conducting periodic reviews to ensure 
that the curriculum is being implemented with fidelity, (4) implementing a new 
school model, (5) providing staff with support and professional development to assist 
both English language learners (ELLs) and students with disabilities, (6) using and 
integrating technology-based supports, and (7) tailoring strategies for secondary 
schools. We used each of those seven subsections as subtopics in our analysis. 

2. Selecting questions aligned with the SIG application criteria. For each subtopic, 
we then used a systematic approach to select survey questions that aligned with the 
practices that SIG sought to promote for that subtopic (by either requiring or 
pennitting them for specific school intervention models). First, a Mathematica 
researcher selected questions corresponding to each practice based on whether he or 
she detennined them to be aligned with the SIG application materials. Another 


18 Regarding how our results might have changed if we had defined the second group differently, if one 
believes that some schools that received SIG funding but didn’t implement a model or some schools that 
implemented a model without SIG funding might have used SIG-promoted practices (for example, because they 
chose to use SIG-promoted practices despite not being required to do so, or because they found a way to use these 
practices without the support of SIG funding), then our estimates of the differences in outcomes between schools 
implementing a SIG-funded model and schools not implementing a SIG-funded model are likely to be 
underestimated (that is, these differences would have been larger if the group of schools not implementing a SIG- 
funded model included only schools that neither received SIG nor implemented a SIG model). 
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Mathematica researcher then reviewed those decisions to ensure that he or she agreed 
with each one (the agreement rate was 100 percent). 19 We detennined the topic area 
and subtopic in which each survey question fell based on the section of the SIG 
application criteria with which it aligned. We did not use a survey question for more 
than one subtopic because that would have resulted in the question being 
overweighted in the overall topic area. When a question could potentially be used for 
more than one subtopic, we assigned it to the subtopic (and corresponding section of 
the application criteria) with which it was best aligned. The survey questions 
addressed all four topic areas and all but 2 of 19 subtopics from the application 
criteria. 

3. Constructing practice variables from survey questions. For each practice in the 
SIG application criteria for which we identified one or more relevant questions, we 
constructed a variable ranging from zero to one using those questions. A value of one 
indicates that the school responded “yes” to all the questions aligned with that 
practice in the application criteria, a value of zero indicates that the school responded 
“yes” to none of the questions aligned with that practice, and a value between those 
two limits indicates that the school responded “yes” to some of the questions aligned 
with that practice. 20 

4. Summing the practices for each school. To detennine each school’s progress in 
using the practices aligned with the SIG application criteria, we summed the variables 
created in step 3. This sum was calculated separately for each subtopic. We then 
summed across subtopics to create a sum for each topic area. 21 If a particular school 
was missing values for a particular practice, we took the mean of the non-missing 
practices and multiplied it by the total number of practices for the overall area. For 
example, for the comprehensive instructional refonn strategies area, which has eight 
practices, if a school had data available for five practices, and reported using two of 
them, the number of the school’s reported used practices would be equal to (2/5)*8. 


19 It was important to have a second researcher review these decisions to ensure that the survey questions 
selected for each subtopic were aligned with the SIG application criteria. Measurement of the extent to which the 
first and second researchers agreed on which questions were aligned with the SIG application criteria is called 
“inter-rater reliability” in statistics. Inter-rater reliability is traditionally measured using the percent agreement rate, 
calculated as the number of questions for which the first and second researchers agreed on whether or not the 
question was aligned with the SIG application criteria, divided by the total number of survey questions (Gwet 2014). 

20 Many questions were originally structured with two response options, with a response of “yes” (recoded to a 
value of one) indicating that the school reported having the practice in place and a response of “no” (recoded to a 
value of zero) indicating that the school did not report having the practice in place. In some cases, however, it was 
necessary to combine multiple survey questions to determine whether or not a school reported frilly adopting a 
particular practice. For example, one practice in the application criteria is that schools use teacher evaluation 
systems that take into account several factors. The survey asked nine separate questions about whether each of nine 
different measures of teacher performance (such as classroom observations and student surveys) was used for 
teacher evaluations. In this case, a school received a fraction of a point (in this example, one-ninth) for each “yes” 
response. This approach helped to ensure that we did not overweight some survey questions relative to how they 
were represented in the application criteria. 

21 Thus, one or more survey questions were used to create a variable for each practice , one or more practices 
formed a subtopic, and one or more subtopics formed a topic area (or area, for short). 
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Across all schools and all subtopics, the average percentage of practices that were 
missing was 6.0 percent. 22 

5. Averaging the number of practices across schools. For each group of schools (that 
is, schools implementing a SIG-funded model and schools not implementing a SIG- 
funded model), we averaged the numbers calculated in step 4. We calculated this 
average number of practices reported for the two groups of schools separately for 
each topic area and subtopic. 

6. Testing differences between groups of schools. We conducted statistical tests to 
assess whether the average number of practices reported differed between the schools 
implementing a SIG-funded model in 2011-2012 and the schools not implementing 
one. 23-24 Because the goal of this analysis was to provide descriptive infonnation 
about the actual levels of practices used by schools implementing a SIG-funded 
model in 2011-2012 and schools not implementing one, the results are reported as 
raw (that is, unadjusted) means; they are not regression-adjusted to account for pre- 
existing differences between these two groups of schools. Throughout this report, we 
focused on the statistical significance of differences between schools implementing a 
SIG-funded model in 2011-2012 and schools not implementing one (rather than the 
magnitude of differences) to ensure that consistent, objective, and transparent criteria 
were used for reporting findings. One caveat with this approach is that some 
statistically significant differences might not be substantively important; we indicated 
places in the report where this might be the case. 

This method of summarizing findings is one way to analyze broad patterns observed in the 
data, and compare levels of usage of practices across different groups of schools. If variables had 
been constructed differently (for example, if multiple questions that addressed the same practice 
had not been combined into a single variable, but had each been included in the analysis as 
separate variables), the results might change. Therefore, it is important to keep these methods in 


22 To assess how much the way we coded missing data might have affected our results, we conducted a 
bounding exercise in which we re-calculated the results twice: once setting all missing responses to “no” (that is, 
assuming all missing responses indicated that the practice was not used) and once setting all missing responses to 
“yes” (that is, assuming all missing responses indicated that the practice was used). The results were largely 
unchanged. The magnitude of differences between schools implementing a SIG-funded model in 2011-2012 and 
schools not implementing one with respect to the number of SIG-promoted practices used were very similar to the 
magnitudes reported in Chapter IV. In addition, across the eight statistical significance tests conducted as part of this 
bounding exercise (two for each of the four topic areas), only one result differed from what is shown in Chapter IV : 
when setting all missing responses to “no,” the difference between schools implementing a SIG-funded model in 
20 1 1-2012 and schools not implementing one with respect to the number of SIG-promoted practices used in the area 
of comprehensive instructional reform strategies was no longer statistically significant. 

23 For this analysis, we used a permutation test, which is the nonparametric counterpart to a t-test. The 
statistical power of this test differed by topic area and subtopic because it depended on several factors, including the 
number of survey questions aligned with the SIG application criteria, the number of variables constructed from those 
questions, and the degree to which the variables were correlated with each other. 

24 There was no need to adjust the standard errors in this analysis for any type of clustering (for example, at the 
district or state level) because there was no mechanism by which districts or states contributed to random variation 
in the results. If we had randomly sampled or randomly assigned districts or states, then it would have been 
appropriate to account for such variation. 
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mind when interpreting the results, along with the caveats mentioned above about the survey 
instrument and the wording of the SIG application criteria. 

3. ELL-Focused Analyses 

We examined the extent to which schools focused on ELLs in their usage of SIG-promoted 
practices using the same processes that we describe above to analyze data from the school 
survey. The only difference was that the summary measures included only practices that were 
explicitly focused on ELLs. We used a systematic approach to select ELL-focused survey 
questions that aligned with the practices in the SIG application criteria, in which two researchers 
selected questions independently (the agreement rate was 100 percent). 

To examine whether usage of these ELL-focused practices differed by the size of the ELL 
population and the ELL/non-ELL achievement gap, we categorized each school according to 
whether it had an above-median or below-median ELL population and an above- or below- 
median ELL/non-ELL achievement gap, where ELL population is defined as the percentage of 
students who are ELLs. We then examined the usage of ELL-focused practices for those groups 
(above-median ELL population, below-median ELL population, above-median gap, below- 
median gap). Throughout this report, we use “schools with higher ELL populations” to refer to 
schools with above-median ELL populations, “schools with higher ELL/non-ELL achievement 
gaps” to refer to schools with above-median ELL/non-ELL achievement gaps, “schools with 
lower ELL populations” to refer to schools with below-median ELL populations, and “schools 
with lower ELL/non-ELL achievement gaps to refer to schools with below-median ELL/non- 
ELL achievement gaps. 

We classified schools as having higher (above-median) or lower (below-median) ELL 
populations using student-level administrative data from 2009-2010, which contained indicators 
for whether each student participated in a program for ELLs. This data element was available for 
all schools in the study sample. We classified schools as having higher or lower ELL/non-ELL 
achievement gaps based on their gaps on the state math assessment, using student-level 
administrative data from 2009-2010. 25 Specifically, we calculated the ELL/non-ELL 
achievement gap as average achievement for non-ELLs minus average achievement for ELLs. 
We used administrative data from 2009-2010 because it was the year prior to the round of SIG 
awards on which we focus in this report. To calculate these variables, we first used student-level 
data to compute the ELL population and the ELL/non-ELL achievement gap for each school in 
our SIG sample. We then used these school-level values to detennine the median ELL 
population and median ELL/non-ELL achievement gap for the schools in our SIG sample. 

For all the ELL-focused analyses of school survey data, we excluded schools that had no 
ELLs, because they were instructed to report “not applicable” for questions that focused on 
ELLs. Therefore, these schools would not have reported infonnation about their practices related 
to ELLs. 


25 For consistency with the state analysis for RTT described in Volume I of this report, we used the math 
assessment, rather than the reading assessment, to calculate achievement gaps. The math and reading gaps were 
highly correlated (0.6 for schools and 0.9 for districts), so the choice of subject was unlikely to make a large 
difference in the composition of the higher and lower groups. 


15 



II. Study Sample, Data Collection, and Analysis 


Usage of Practices Promoted by SIG 


We ran statistical tests to detennine whether there were differences in the number of ELL- 
focused practices used between each group of schools. For example, we compared SIG-funded 
model schools with lower ELL populations and ELL/non-ELL achievement gaps to schools that 
also had lower populations and gaps but did not implement such a model. We also ran statistical 
tests to detennine whether there were differences in the number of ELL-focused practices within 
each group of schools. For example, we compared SIG-funded model schools with lower 
populations and gaps to SIG-funded model schools with higher populations and gaps. 
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III. UNDERSTANDING THE CONTEXT FOR USAGE OF PRACTICES PROMOTED 

BY SCHOOL IMPROVEMENT GRANTS 

In this chapter, we examine baseline characteristics of the two groups of schools compared 
in this volume of the report: (1) schools implementing a SIG-funded intervention model in 20 11— 
2012 and (2) schools not implementing a SIG-funded model in 2011-2012. These schools were 
purposively selected to support the estimation of impacts of SIG-funded models on student 
outcomes that will be presented in a future report for this evaluation. Because the SIG sample 
was not randomly selected, findings cannot necessarily be generalized to schools implementing a 
SIG-funded model nationwide, and readers should exercise caution when interpreting the results. 
However, given the limited information currently available about the implementation of SIG, the 
implementation findings presented in this report are still relevant for the SIG program. 

To help contextualize study findings, it is important to understand differences and 
similarities that may have existed at baseline (i.e., during the 2009-2010 school year, which is 
the year prior to the round of SIG awards that we focus on in this study) between the two groups 
of schools in our analysis. The characteristics of study schools implementing and not 
implementing SIG-funded models are shown in Table III. 1 (first and second columns). A 
comparison of these first two columns suggests that our two groups of study schools were 
generally similar at baseline, with a few exceptions. Study schools implementing a SIG-funded 
model had a statistically significantly higher percentage of students who were eligible for free or 
reduced-price lunch than study schools not implementing a SIG-funded model (83.5 vs. 80.4 
percent). Study schools implementing a SIG-funded model also were unsurprisingly more likely 
to be implementing one of the four intervention models prescribed by SIG, to be eligible for SIG 
under Tier I or II (which were prioritized for SIG awards over Tier III schools), and to be in SIG 
cohort 1 or cohort 2. 

To understand how the study schools implementing a SIG-funded model in 2011-2012 
compare to all schools in the United States that were implementing SIG-funded models in 20 11— 
2012, data on the latter group are shown in the third column of Table III. 1 . Comparing the first 
and third columns shows that study schools implementing a SIG-funded model differed from 
U.S. schools implementing such models on nearly all of the baseline measures examined. For 
example, study schools implementing a SIG-funded model were significantly more likely to be 
economically disadvantaged (with 83.5 vs. 78.2 percent of students eligible for free or reduced- 
price lunch), located in an urban area (87.3 vs. 59.1 percent), and to use the turnaround model 
(36.3 vs. 21.3 percent). These differences are perhaps not surprising, given that (as described in 
Chapter II), the schools in the SIG sample were more likely than other schools to be located in 
large districts. 

These patterns suggest that the two groups of study schools in our SIG sample are generally 
similar, but that the sample of study schools implementing a SIG-funded model is not 
representative of such schools nationwide. Therefore, readers should exercise caution when 
interpreting the results. In particular, it should not be assumed that SIG implementation findings 
presented in this report necessarily generalize to SIG schools nationwide. 
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Table 111.1. Baseline (2009-2010) Characteristics of Study Schools and of All U.S. Schools Implementing a 
SIG-Funded Intervention Model 



Study Schools 

Study Schools Not 

All U.S. Schools 


Implementing a SIG- 

Implementing a SIG- 

Implementing a SIG- 


Funded Intervention 

Funded Intervention 

Funded Intervention 


Model in 2011-2012 

Model in 2011-2012 

Model in 2011-2012 

Average Percentage of Students by 
Race/Ethnicity Category 

White, non-Hispanic 

9.3* 

8.9 

19.0 

Black, non-Hispanic 

54.2* 

47.9 

45.4 

Hispanic 

31.3* 

37.2 

26.7 

Asian 

1.9 

2.0 

2.2 

Other 

3.3* 

3.9 

6.7 

Average Percentage of Students Eligible for 

Free or Reduced-Price Lunch 

83.5t* 

80.4 

78.2 

Percentage of Title 1 Eligible Schools 

94.8* 

93.3 

89.2 

Percentage of Schools by Location 

Urban 

87.3* 

86.6 

59.1 

Suburban 

6.5* 

6.1 

16.0 

Town or Rural 

6.2* 

7.3 

24.9 

Percentage of Schools by Level 

Elementary 

29.6* 

30.2 

24.0 

Middle 

19.9 

17.9 

19.5 

High 

48.1 

48.0 

49.4 

Other 

2.4* 

3.9 

7.1 

Percentage of Schools by Model 

Turnaround 

36. 3f* 

0.0 

21.3 

Restart or Closure 3 

6.8| 

0.0 

5.9 

Transformation 

56.8t* 

0.0 

72.8 

Percentage of Schools by SIG Cohort 

Cohort 1 

86.3t* 

2.2 b 

46.8 

Cohort 2 

13. 7f* 

0.0 

53.3 

Cohort 3 

O.Of 

5.6 

0.0 

Percentage of Schools by Eligibility Tier 

Tier 1 

64. 8f 

13.9 

62.7 

Tier II 

24. 7f* 

8.5 

37.3 

Tier III 

10.5t 

77.6 

n.a. 

Number of Schools 

290 

170-180 

830-840 


Source: Common Core of Data, 2009-2010; IES database of SIG grantees; surveys of school administrators in 

spring 2012; interviews with district administrators in spring 2012. 

Note: Percentages of students are unweighted school-level averages. Study schools identified as 

implementing or not implementing a SIG-funded intervention model were identified using information 
from districts and schools, as well as the IES database of SIG grantees available at 
http://www2.ed.gov/programs/sif/index.html. These data sources were also used to identify the 
particular intervention model being implemented by study schools. U.S. schools implementing a SIG- 
funded intervention model (and the particular intervention model being implemented by these schools) 
were identified using the IES database of SIG grantees, and this group of schools was restricted to 
schools in Tiers I and II because ED required that each Tier I or II school receiving SIG implement one 
of four school intervention models (whereas Tier III schools receiving SIG were not required to do so), 
so schools in Tiers I and II are more similar to the group of study schools implementing a SIG-funded 
intervention model than Tier III schools are. The national percentages of schools implementing each of 
the four intervention models are based on schools’ planned implementation as of 2009-2010 for cohort 
1 grantees and as of 2010-2011 for cohort 2 grantees. Data from 2009-2010 are used whenever 
possible to report schools’ demographic and location data because that was the school year just before 
the first year of implementation of the ARRA-funded SIG intervention models. Data from 2008-2009 are 
used for schools with data missing in 2009-2010, and data from 2007-2008 are used for schools with 
data missing in both 2009-2010 and 2008-2009. National comparison data are for Tier I and II schools 
in 49 states and the District of Columbia. One state, Hawaii, is excluded in the national comparison data 
because the SIG database does not include information for Hawaii. To comply with NCES statistical 
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Table III. 1 (continued) 

reporting requirements for small cell sizes, we aggregated the percentages for town and rural school 

locations and for restart and closure intervention models. 

a Schools that had already implemented the closure model as of spring 2012 were not surveyed and were not included 
in the analysis. Schools that were planning to implement the closure model but had not yet closed as of spring 2012 
were surveyed and included in the analysis, for three reasons: (1) Dropping these schools from the analysis would have 
been inconsistent with how we treated schools that were planning to implement other models, but had not yet 
implemented a particular practice required by that model. For example, if a school was planning to implement the 
transformation model but had not yet replaced their principal, we still treated them as schools implementing a SIG- 
funded intervention model in 201 1-2012. Similarly, we treated schools that were planning to close but had not yet closed 
as of spring 2012 as schools implementing a SIG-funded intervention model in 2011-2012; (2) The SIG application 
guidance indicated that closure schools may use SIG funds to cover the activities (such as community outreach) that 
were recommended before closing the school. Because these schools were receiving SIG funds to implement the 
closure model and the associated activities that preceded the closure, we included them in the analysis for the years 
before they closed; and (3) The process of closure was not always immediate: some schools closed by allowing current 
students to finish, but ending enrollment of additional students (that is, the lowest grade closed first, then the next lowest, 
and so on, until the school was shut down). 

b Cohort 1 schools that were not implementing a SIG-funded intervention model in 2011-2012 are schools that replied 
“no” to either question TA1 on the 2012 school survey (which asked whether the school received SIG funds for school 
improvement efforts in the current school year), or question TA7 (which asked if the school was using one of the four 
ED-specified intervention models), or both. (Note that a school had to reply “yes” to both questions to be considered 
implementing a SIG-funded intervention model in our analysis.) For the schools for which we had information from 
another source (specifically, the district interview), that source corroborated the information provided in the school 
survey. 

tSignificantly different from study schools not implementing a SIG-funded intervention model in 2011-2012 at the 0.05 
level, two-tailed test. 

‘Significantly different from schools in the U.S. implementing a SIG-funded intervention model in 201 1-2012 at the 0.05 
level, two-tailed test. 

IES = Institute of Education Sciences; n.a. = not applicable. 

We also compared (1) the characteristics of states in which our sample of SIG schools are 
located to all states in the United States (Table III. 2) and (2) the characteristics of the districts 
with schools in our SIG sample to all districts in the United States with at least one school 
implementing a SIG-funded model (Table III. 3). Because all states in the United States contained 
schools implementing a SIG-funded intervention model in 2011-2012, the group of all states is 
an appropriate comparison group for the states in our SIG sample. The characteristics of states 
with schools in our SIG sample did not differ significantly from all states, indicating that the 
states included in our SIG sample are likely representative of all states that contained schools 
implementing a SIG-funded intervention model in 2011-2012. The SIG-sample districts differed 
from all districts with schools implementing a SIG-funded model in terms of students’ race and 
school location. For example, the districts in our study had a higher percentage of students who 
were non-Hispanic black (38.7 vs. 30.3 percent) and had schools that were more likely to be 
located in an urban area (68.2 vs. 37.7 percent). 

Table 111.2. Baseline (2009-2010) Characteristics of the SIG-Sample States and All States 


SIG-Sample States All States' 1 


Percentage of Students in the Following 
Race/Ethnicity Categories: 


White, non-Hispanic 

55.3 

61.8 

Black, non-Hispanic 

19.5 

15.8 

Hispanic 

18.3 

13.7 

Asian 

3.8 

4.6 

Other 

3.1 

4.1 

Percentage of Students Eligible for Free or 
Reduced-Price Lunch 

48.0 

45.5 
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Table III. 2 (continued) 



SIG-Sample States 

All States 3 

Percentage of Schools That Are Title 1 
Eligible 

68.1 

67.8 

Percentage of Schools in the Following 
Locations: 



Urban 

30.0 

23.3 

Suburban 

25.7 

22.5 

Town 

14.3 

16.0 

Rural 

30.0 

38.2 

Number of States 

22 

51 


Source: Common Core of Data, 2009-2010. 

Note: Data from 2008-2009 are used for states with data missing in 2009-2010. Data from 2007- 

2008 are used for states with data missing in both 2009-2010 and 2008-2009. Data from 
2009-2010 are used whenever possible because that was the school year just before the first 
year of implementation of the ARRA-funded SIG intervention models. Percentages of 
students and schools are unweighted state-level averages. There were no statistically 
significant differences between SIG-sample states and all states at the 0.05 level using a two- 
tailed test. 

a Includes 50 states and the District of Columbia, all of which contained schools implementing a SIG-funded 
intervention model in 2011-2012. 

Table III. 3. Baseline (2009-2010) Characteristics of the SIG-Sample Districts and All U.S. Districts 
with Schools Implementing a SIG-Funded Intervention Model 


Districts in the U.S. 

With at Least One School 
Implementing a SIG-Funded 
Intervention Model 



SIG-Sample Districts 

in 2011-2012 

Percentage of Students in the Following 
Race/Ethnicity Categories: 

White, non-Hispanic 

19.5* 

33.4 

Black, non-Hispanic 

38.7* 

30.3 

Hispanic 

32.0 

25.8 

Asian 

3.3 

2.5 

Other 

6.5 

8.0 

Percentage of Students Eligible for Free 

or Reduced-Price Lunch 

72.4 

68.1 

Percentage of Schools That Are Title 1 

Eligible 

81.4 

83.0 

Percentage of Districts in the Following 
Locations: 

Urban 

68.2* 

37.7 

Suburban 

17.3 

20.0 

Town 

5.7 

12.5 

Rural 

8.8* 

29.8 

Number of Districts 

60 

610 


Source: Common Core of Data, 2009-2010; IES database of SIG grantees. 

Note: Data from 2008-2009 are used for districts with data missing in 2009-2010. Data from 2007- 

2008 are used for districts with data missing in both 2009-2010 and 2008-2009. Data from 
2009-2010 are used whenever possible because that was the school year just prior to the 
first year of implementation of the ARRA-funded SIG intervention models. Districts in the 
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U.S. with at least one school implementing a SIG-funded intervention model in 2011-2012 
were identified using information from the IES database of SIG grantees available at 
http://www2.ed.gov/programs/sif/index.html. Percentages of students and schools are 
unweighted district-level averages. Comparison data are for districts in 49 states and the 
District of Columbia because the SIG database does not include information for Hawaii. The 
percentages of districts with at least one school implementing a SIG-funded intervention 
model are based on schools’ planned implementation as of 2009-2010 for cohort 1 grantees, 
as of 2010-2011 for cohort 2 grantees, and only include Tier I and Tier II schools. 
*Significantly different from districts in the U.S. with at least one school implementing a SIG-funded 
intervention model in 2011-2012 at the 0.05 level, two-tailed test. 
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IV. SCHOOLS’ USAGE OF PRACTICES PROMOTED BY 
SCHOOL IMPROVEMENT GRANTS 

To understand the effectiveness of a grant program like SIG, it is important to first 
understand the extent to which the practices it promotes are being used by low-perfonning 
schools that implemented a SIG-funded intervention model. If none (or very few) schools have 
used these practices, it is unlikely that any changes in outcomes — positive or negative — could be 
attributed to the program. In this chapter, we assess the extent to which schools implementing a 
SIG-funded model and those not implementing one reported using practices promoted by SIG. 

As noted previously, SIG promoted the implementation of four school intervention models, 
which prescribed specific practices in four topic areas. In this chapter, we summarize the extent 
to which schools reported in spring 2012 using practices aligned with each of these topic areas 
and their subtopics. 26 Appendix A presents more detailed findings for the subtopics. As noted in 
Chapters II and III, the findings should be interpreted with caution because the SIG sample was 
not randomly selected and is not representative of schools nationwide. 

While schools use practices promoted by SIG, districts might also use practices promoted by 
SIG to support schools implementing a SIG-funded model. For example, districts might require 
multiple performance measures for teacher and principal evaluations or provide additional 
supports and programs to students with disabilities. To provide context for school reports about 
practices aligned with the SIG application criteria, Appendix B displays findings on the extent to 
which districts reported using the practices promoted by SIG. As noted in Chapters II and III, the 
findings should be interpreted with caution because the SIG sample was not randomly selected 
and is not representative of districts nationwide. 

A. Comprehensive Instructional Reform Strategies 

One of the goals of SIG is to promote the usage of instructional practices that have the 
potential to increase academic rigor and achievement of students. The SIG application criteria 
focused on practices to reform instruction in seven subtopics: (1) using data to identify and 
implement an instructional program, (2) promoting the continuous use of data to identify and 
address the needs of individual students, (3) conducting periodic reviews of the curriculum, 
(4) implementing a new school model, (5) providing supports and professional development 
(PD) to staff in order to assist both ELLs and students with disabilities, (6) using and integrating 
technology based supports, and (7) tailoring strategies for secondary schools. 27 We identified 
eight practices from the spring 2012 school administrator survey aligned with SIG objectives on 
comprehensive instructional reform strategies (Table IV. 1). 


26 The analyses presented in this report do not distinguish between required and permissible practices; 
according to the SIG application criteria, required practices are those that schools implementing a particular SIG 
model must use, and permissible practices are those that schools implementing a particular SIG model may use. For 
a detailed examination of whether low-performing schools adopted the practices that were required and/or 
permissible under the transformation and turnaround models, please see Herrmann, M., L. Dragoset, and S. James- 
Burdumy. “Are Low-Performing Schools Adopting Practices Promoted by School Improvement Grants?” NCEE 
2015-4001. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for 
Education Evaluation and Regional Assistance, October 2014. 

27 Because no school administrator survey items aligned to subtopics 3 and 4, they are excluded from the 
analysis. 
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Table IV.1. Practices Aligned with SIG Objectives on Comprehensive Instructional Reform Strategies, by 
Subtopic 


Using Data to Identify and Implement an Instructional Program 

Using data to evaluate instructional programs (for example, measuring program effectiveness) 

Promoting the Continuous Use of Student Data 

Using data to inform and differentiate instruction 

The typical English/language arts or math teacher used benchmark or interim assessments at least once per year 

Providing Supports and Professional Development to Staff to Assist ELLs and Students with Disabilities 

Implementing strategies (including additional supports or professional development) to ensure that limited English 
proficient students acquire language skills to master academic content 

Using and Integrating Technology-Based Supports 

Increased access to technology for teachers or that the typical English/language arts teacher used computer- 
assisted instruction 


Tailoring Strategies for Secondary Schools 

Secondary school monitored students’ college readiness (such as enrollment in Advanced Placement courses), 
including providing supports (such as project-based learning) so that low-achieving students can take advantage 
of these types of opportunities 3 

The school or grades within the secondary school were subdivided into small learning communities or field/career- 
oriented academies 3 

Secondary school tracked student progress towards (and readiness for) high school graduation 3 
Source: SIG application; surveys of school administrators in spring 2012. 

Note: See Appendix D for a list of the specific survey questions that were aligned with the SIG practices in this 

table. 

3 As described in Chapter II, to deal with the fact that a particular school (in this case, an elementary school) might 
have missing values for a particular practice, we took the mean of the non-missing practices and multiplied it by the 
total number of practices for the overall topic area. For example, for the comprehensive instructional reform strategies 
topic area, which has eight practices, if a school had data available for five practices, and reported using two of those, 
the number of the school’s reported used practices would be equal to (2/5)*8. 

ELLs = English language learners. 

In spring 2012, schools implementing a SIG-funded model reported using statistically 
significantly more of the practices aligned with the comprehensive instructional reform 
criteria in the SIG application than did schools not implementing a SIG-funded model. 

Schools implementing a SIG-funded model reported using an average of 7.3 of 8 practices in this 
area, compared to 7.0 for schools not implementing such a model (Figure IV. 1), a difference of 
0.3 practices. Therefore, although this difference was statistically significant, it may not be 
substantively important. 
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Figure IV.1. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on Comprehensive 
Instructional Reform Strategies, Spring 2012 



Schools Implementing a SIG-Funded 
Intervention Model in 2011-2012 
(N=290) 


Schools Not Implementing a SIG-Funded 
Intervention Model in 2011-2012 
(N=180) 


Source: Surveys of school administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table IV.1. Each dot in this figure represents the schools 

that reported using a particular number of practices (out of eight examined) that were aligned with the SIG 
application criteria. The number inside each dot is the number of schools represented by the dot; dots that represent 
less than 10 schools have no number inside. For example, 24 schools implementing a SIG-funded intervention 
model reported using seven of the eight comprehensive instructional reform practices aligned with the SIG 
application criteria. For two of the practices, a “yes” response received one point. In the other six cases, it was 
possible for a school to receive a fraction of one point. See Chapter II for details on the way in which the number of 
practices was determined for each school. The dashed line denotes the average number of practices for each group 
of schools. 

‘Significantly different from schools not implementing a SIG-funded intervention model at the 0.05 level, two-tailed test. 


24 


IV. Schools ’ Usage of Practices Promoted by SIG 


Usage of Practices Promoted by SIG 


In spring 2012, schools implementing a SIG-funded model reported using statistically 
significantly more of the practices aligned with the SIG application criteria in two of five 
comprehensive instructional reform subtopics than did schools not implementing a SIG- 
funded model: 

• Promoting the continuous use of student data. Schools implementing a SIG-funded 
model reported using an average of 1 .94 of 2 practices in this subtopic, compared to 
1.90 for schools not implementing such a model (Appendix A Figure A.2). 

• Using and integrating technology-based supports. Ninety-one percent of schools 
implementing a SIG-funded model reported using the practice in this subtopic, 
compared to 77 percent of schools not implementing such a model (Appendix A 
Figure A. 4). 

The individual practice with the highest level of usage in the comprehensive 
instructional reform strategies area was using benchmark or interim assessments at least 
once per year. Across all study schools, the average value for this practice was 0.99. 28 

Average values for each group of schools (those implementing a SIG-funded intervention model 
and those not implementing one) are shown in Figure IV. 2. 

The individual practice with the lowest level of usage in the comprehensive 
instructional reform strategies area was subdividing the secondary school or grades within 
it into small learning communities or field/career-oriented academies. Across all study 
schools, the average value for this practice was 0.62. Average values for each group of schools 
(those implementing a SIG-funded intervention model and those not implementing one) are 
shown in Figure IV. 2. 


28 As described in Chapter II, for each practice in the SIG application criteria for which we identified one or 
more relevant survey questions, we constructed a variable ranging from zero to one using those questions. To 
calculate the average value of a practice, we averaged that variable across all study schools. For example, if half the 
schools adopted a particular practice (meaning they had a value of 1) and the other half of schools did not use that 
practice (meaning they had a value of 0), the average value for that practice would be 0.5. 
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Figure IV.2. Study Schools’ Reported Usage of Individual Practices Aligned with SIG Objectives on 
Comprehensive Instructional Reform Strategies, Spring 2012 
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Practice Aligned with SIG Objectives 

Schools Implementing a SIG-Funded Intervention Model in 2011-2012 (N=1 30-290) 
Schools Not Implementing a SIG-Funded Intervention Model in 2011-2012 (N=80-180) 


Source: Surveys of school administrators in spring 2012. 

Note: This figure has a separate panel for each subtopic. As described in Chapter II, for each practice in the 

SIG application criteria for which we identified one or more survey questions aligned with the practice, 
we constructed a variable ranging from zero to one, with a value of one indicating that the school 
responded “yes” to all the survey questions selected for that practice. The height of each bar represents 
the mean value of the practice (on a scale of zero to one) for each group of schools. For some of the 
practices shown in this figure, multiple survey questions aligned with that practice. In the figure, we 
indicate this using the words “extent to which” at the beginning of the practice, to emphasize that the 
level of usage of that practice is measured using multiple survey questions (as opposed to a single, 
binary measure of whether that practice was used). A range is provided for the sample sizes because 
nonresponse varied across items. 

ELLs = English language learners; PD = professional development. 
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B. Teacher and Principal Effectiveness 

The SIG program encouraged schools and districts receiving SIG to increase the capacity 
and quality of their teachers and principals to improve student outcomes. Specifically, the SIG 
application criteria focused on practices to develop and increase teacher and principal 
effectiveness in four subtopics: (1) using rigorous, transparent, and equitable evaluation systems; 

(2) identifying and rewarding effective teachers and principals and removing ineffective ones; 

(3) providing high-quality, job-embedded professional development or supports; and 

(4) implementing strategies to recruit, place, and retain staff. We identified 20 practices from the 
spring 2012 school administrator survey aligned with SIG objectives on developing and 
increasing teacher and principal effectiveness (Table IV.2). 

Table IV.2. Practices Aligned with SIG Objectives on Teacher and Principal Effectiveness, by Subtopic 


TEACHER EFFECTIVENESS 


Using Rigorous, Transparent, and Equitable Evaluation Systems 

Student achievement growth was a required component of teacher evaluations, and the extent to which student 
achievement growth must factor into teacher evaluations or that state test scores were used to assess student 
growth for teacher evaluations was specified 

Using multiple performance measures for teacher evaluations 

Identifying and Rewarding Effective Teachers and Removing Ineffective Ones 

Using teacher evaluation results to inform decisions about compensation 

Reviewing the strengths and competencies of instructional staff for the purposes of hiring or removing staff 

Providing High-Quality, Job-Embedded Professional Development or Supports 

Providing instructional staff with PD that consisted mostly or entirely of multiple-session events 

Providing instructional staff with PD that focused on transitioning to Common Core State Standards, aligning 
instruction to state standards, or strategies for turning around a low-performing school 

Providing staff with PD that involved educators working collaboratively or was facilitated by school leaders or 
coaches 

Providing staff with PD that was focused on understanding and addressing student learning needs (including 
reviewing student work and achievement data, and collaboratively planning, testing, and adjusting instructional 
strategies based on data) 

Providing staff with PD designed with input from school staff 

Using data to evaluate the success of PD offerings 

Implementing Strategies to Recruit, Place, and Retain Staff 

Implementing strategies, such as financial incentives or more flexible work conditions, that were designed to 
recruit, place, and retain staff 

Using teacher evaluation results as the primary consideration in reductions in force and excessing decisions, or 
having teacher assignment policies that allow for principal discretion to decide which staff to hire for the school 

PRINCIPAL EFFECTIVENESS 


Using Rigorous, Transparent, and Equitable Evaluation Systems 

Measures of student achievement growth were used for principal evaluations and the extent to which student 
achievement growth must factor into principal evaluations was specified 

Using multiple performance measures for principal evaluations 

Identifying and Rewarding Effective Principals and Removing Ineffective Ones 

Using principal evaluation results to inform decisions about compensation 
School has a new principal 
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PRINCIPAL EFFECTIVENESS 

Providing High-Quality, Job-Embedded Professional Development or Supports 

State or district provides the principal or other school leaders with PD on analyzing and revising budgets or 
strategies for turning around a low-performing school 

State or district provides the principal or other school leaders with PD on identifying effective instructional staff for 
leadership positions and supporting them in these positions 

State or district uses principal evaluation results to develop the principal’s PD or provides the principal with PD on 
aligning teachers’ PD with evaluation results 

Implementing Strategies to Recruit, Place, and Retain Staff 

Principals have the opportunity to receive financial incentives designed to recruit, place, and retain staff 
Source: SIG application; surveys of school administrators in spring 2012. 

Note: See Appendix D for a list of the specific survey questions that were aligned with the SIG practices in this 

table. 

PD = professional development. 

In spring 2012, schools implementing a SIG-funded model reported using statistically 
significantly more practices aligned with the teacher and principal effectiveness criteria in 
the SIG application than did schools not implementing a SIG-funded model. Schools 
implementing a SIG-funded model reported using an average of 1 1.1 of 20 practices in this area, 
compared to 9.6 for schools not implementing such a model (Figure IV. 3), a difference of 1.5 
practices. Therefore, although this difference was statistically significant, it may not be 
substantively important. 

In spring 2012, schools implementing a SIG-funded model reported using statistically 
significantly more of the practices aligned with the SIG application criteria in two of the 
four teacher and principal effectiveness subtopics than did schools not implementing a 
SIG-funded model: 

• Identifying and rewarding effective teachers and principals and removing 
ineffective ones. Schools implementing a SIG-funded model reported using an 
average of 1.5 of 4 practices in this subtopic, compared to 1.2 for schools not 
implementing such a model (Appendix A Figure A. 7). 

• Providing high-quality, job-embedded professional development or supports. 

Schools implementing a SIG-funded model reported using an average of 6.4 of 9 
practices in this subtopic, compared to 5.6 for schools not implementing such a model 
(Appendix A Figure A. 8). 

The individual practice with the highest level of usage in the teacher and principal 
effectiveness area was providing staff with professional development focused on 
understanding and addressing student learning needs. Across all study schools, the average 
value for this practice was 0.88. Average values for each group of schools (those implementing a 
SIG-funded intervention model and those not implementing one) are shown in Figure IV. 5. 

The individual practice with the lowest level of usage in the teacher and principal 
effectiveness area was using teacher evaluation results to inform decisions about 
compensation. Across all study schools, the average value for this practice was 0.15. Average 
values for each group of schools (those implementing a SIG-funded intervention model and 
those not implementing one) are shown in Figure IV. 4. (Figure IV. 6 shows similar infonnation 
for other practices.) 
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Figure IV.3. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on Teacher and 
Principal Effectiveness, Spring 2012 



Schools Implementing a SIG-Funded Schools Not Implementing a SIG-Funded 
Intervention Model in 2011-2012 Intervention Model in 2011-2012 

(N=290) (N=1 80) 


Source: Surveys of school administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table IV. 2. Each dot in this figure represents 

the number of schools that reported using a particular number of practices (out of 20 examined) that 
were aligned with the SIG application criteria. The number inside each dot is the number of schools 
represented by the dot; dots that represent less than 10 schools have no number inside. For example, 
14 schools implementing a SIG-funded intervention model reported using 8 of the 20 teacher and 
principal effectiveness practices aligned with the SIG application criteria. For 15 of the practices, a “yes” 
response received one point. In the other 5 cases, it was possible for a school to receive a fraction of 
one point. See Chapter II for details on the way in which the number of practices was determined for 
each school. The dashed line denotes the average number of practices for each group of schools. 
"Significantly different from schools not implementing a SIG-funded intervention model at the 0.05 level, two-tailed 
test. 
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Figure IV.4. Study Schools’ Reported Usage of Individual Practices Aligned with SIG Objectives on 
Teacher and Principal Effectiveness; Using Rigorous, Transparent, and Equitable Evaluation 
Systems Subtopic and Identifying and Rewarding Effective Teachers and Principals and 
Removing Ineffective Ones Subtopic, Spring 2012 
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Schools Implementing a SIG-Funded Intervention Model in 2011-2012 (N=180-280) 
Schools Not Implementing a SIG-Funded Intervention Model in 2011-2012 (N=120— 1 70) 


Source: Surveys of school administrators in spring 2012. 

Note: This figure has a separate panel for each subtopic. As described in Chapter II, for each practice in the 

SIG application criteria for which we identified one or more survey questions aligned with the practice, 
we constructed a variable ranging from zero to one, with a value of one indicating that the school 
responded “yes” to all the survey questions selected for that practice. The height of each bar represents 
the mean value of the practice (on a scale of zero to one) for each group of schools. For some of the 
practices shown in this figure, multiple survey questions aligned with that practice. In the figure, we 
indicate this using the words “extent to which” at the beginning of the practice, to emphasize that the 
level of usage of that practice is measured using multiple survey questions (as opposed to a single, 
binary measure of whether that practice was used). A range is provided for the sample sizes because 
nonresponse varied across items. 
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Figure IV.5. Study Schools’ Reported Usage of Individual Practices Aligned with SIG Objectives on 
Teacher and Principal Effectiveness; Providing High-Quality, Job-Embedded Professional 
Development or Supports Subtopic, Spring 2012 
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■ Schools Implementing a SIG-Funded Intervention Model in 2011-2012 (N=270-290) 
Schools Not Implementing a SIG-Funded Intervention Model in 2011-2012 (N=1 60— 1 80) 


Source: Surveys of school administrators in spring 2012. 

Note: As described in Chapter II, for each practice in the SIG application criteria for which we identified one or 

more survey questions aligned with the practice, we constructed a variable ranging from zero to one, 
with a value of one indicating that the school responded “yes” to all the survey questions selected for 
that practice. The height of each bar represents the mean value of the practice (on a scale of zero to 
one) for each group of schools. For some of the practices shown in this figure, multiple survey questions 
aligned with that practice. In the figure, we indicate this using the words “extent to which” at the 
beginning of the practice, to emphasize that the level of usage of that practice is measured using 
multiple survey questions (as opposed to a single, binary measure of whether that practice was used). 
A range is provided for the sample sizes because nonresponse varied across items. 

CCSS = Common Core State Standards; PD = professional development. 
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Figure IV.6. Study Schools’ Reported Usage of Individual Practices Aligned with SIG Objectives on 
Teacher and Principal Effectiveness; Implementing Strategies to Recruit, Place, and Retain Staff 
Subtopic, Spring 2012 
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■ Schools Implementing a SIG-Funded Intervention Model in 2011-2012 (N=260-290) 
Schools Not Implementing a SIG-Funded Intervention Model in 201 1-2012 (N=1 60—1 70) 


Source: Surveys of school administrators in spring 2012. 

Note: As described in Chapter II, for each practice in the SIG application criteria for which we identified one or 

more survey questions aligned with the practice, we constructed a variable ranging from zero to one, 
with a value of one indicating that the school responded “yes” to all the survey questions selected for 
that practice. The height of each bar represents the mean value of the practice (on a scale of zero to 
one) for each group of schools. For some of the practices shown in this figure, multiple survey questions 
aligned with that practice. In the figure, we indicate this using the words “extent to which” at the 
beginning of the practice, to emphasize that the level of usage of that practice is measured using 
multiple survey questions (as opposed to a single, binary measure of whether that practice was used). 
A range is provided for the sample sizes because nonresponse varied across items. 

RIF = reductions in force. 


C. Learning Time and Community-Oriented Schools 

To ensure that SIG schools have sufficient time for instruction and a supportive environment 
in which to implement policies, the SIG application criteria focused on practices in two 
subtopics: (1) increasing learning time; and (2) engaging families and communities and 
providing a safe school environment that meets students’ social, emotional, and health needs. We 
identified five practices from the spring 2012 school administrator survey aligned with SIG 
objectives on increasing learning time and creating community-oriented schools (Table IV. 3). 
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Table IV.3. Practices Aligned with SIG Objectives on Learning Time and Community-Oriented Schools, by 
Subtopic 

Increasing Learning Time 

Using schedules and strategies that provide increased learning time or increasing the number of hours per year 
that school was in session 

Engaging Families and Communities and Providing a Safe School Environment that Meets Students’ 

Social, Emotional, and Health Needs 

Changing policies or strategies related to parent or community engagement 

State or district provided professional development on working with parents or creating a safe school environment 

Changing discipline policies 

Guiding the development and implementation of, or making changes to, nonacademic supports or enrichment 
programs for students 

Source: SIG application; surveys of school administrators in spring 2012. 

Note: See Appendix D for a list of the specific survey questions that were aligned with the SIG practices in this 

table. 

In spring 2012, schools implementing a SIG-funded model reported using statistically 
significantly more of the practices aligned with the increasing learning time and creating 
community-oriented schools criteria in the SIG application than did schools not 
implementing a SIG-funded model. Schools implementing a SIG-funded model reported using 
an average of 3.9 of 5 practices in this area, compared to 3.2 for schools not implementing such a 
model (Figure IV. 7), a difference of 0.7 practices. Therefore, although this difference was 
statistically significant, it may not be substantively important. 

In spring 2012, schools implementing a SIG-funded model reported using statistically 
significantly more of the practices aligned with the SIG application criteria in both of the 
increasing learning time and creating community-oriented schools subtopics: 

• Increasing learning time. Sixteen percent of schools implementing a SIG-funded 
model reported using the practice in this subtopic, compared to 8 percent of schools 
not implementing such a model (Appendix A Figure A. 10). 

• Engaging families and communities and providing a safe school environment 
that meets students’ social, emotional, and health needs. Schools implementing a 
SIG-funded model, reported using an average of 3.3 of 4 practices in this subtopic, 
compared to 2.7 for schools not implementing such a model (Appendix A Figure 
A.l 1). 

The individual practice with the highest level of usage in the learning time and 
community-oriented schools area was changing policies or strategies related to parent or 
community engagement. Across all study schools, the average value for this practice was 0.82. 
Average values for each group of schools (those implementing a SIG-funded intervention model 
and those not implementing one) are shown in Figure IV. 8. 

The individual practice with the lowest level of usage in the learning time and 
community-oriented schools area was using schedules and strategies that provide increased 
learning time or increasing the number of hours per year that school was in session. Across 
all study schools, the average value for this practice was 0.58. Average values for each group of 
schools (those implementing a SIG-funded intervention model and those not implementing one) 
are shown in Figure IV. 8. 
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Figure IV.7. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on Learning Time and 
Community-Oriented Schools, Spring 2012 
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(N=290) 


(N=180) 


Source: Surveys of school administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table IV. 3. Each dot in this figure represents 

the number of schools that reported using a particular number of practices (out of five examined) that 
were aligned with the SIG application criteria. The number inside each dot is the number of schools 
represented by the dot; dots that represent less than 10 schools have no number inside. For example, 
25 schools implementing a SIG-funded intervention model reported using just over four of the five 
learning time and community-oriented schools practices aligned with the SIG application criteria. For 
four of the practices, a “yes” response received one point. In the other case, it was possible for a school 
to receive a fraction of one point. See Chapter II for details on the way in which the number of practices 
was determined for each school. The dashed line denotes the average number of practices for each 
group of schools. 

"Significantly different from schools not implementing a SIG-funded intervention model at the 0.05 level, two-tailed 
test. 
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Figure IV.8. Study Schools’ Reported Usage of Individual Practices Aligned with SIG Objectives on 
Learning Time and Community-Oriented Schools, Spring 2012 
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■ Schools Implementing a SIG-Funded Intervention Model in 2011-2012 (N=270-290) 
Schools Not Implementing a SIG-Funded Intervention Model in 2011-2012 (N=1 60— 1 80) 


Source: Surveys of school administrators in spring 2012. 

Note: This figure has a separate panel for each subtopic. As described in Chapter II, for each practice in the 

SIG application criteria for which we identified one or more survey questions aligned with the practice, 
we constructed a variable ranging from zero to one, with a value of one indicating that the school 
responded “yes” to all the survey questions selected for that practice. The height of each bar represents 
the mean value of the practice (on a scale of zero to one) for each group of schools. For some of the 
practices shown in this figure, multiple survey questions aligned with that practice. In the figure, we 
indicate this using the words “extent to which” at the beginning of the practice, to emphasize that the 
level of usage of that practice is measured using multiple survey questions (as opposed to a single, 
binary measure of whether that practice was used). A range is provided for the sample sizes because 
nonresponse varied across items. 

PD = professional development. 


D. Operational Flexibility and Support 

To facilitate the implementation of turnaround efforts and ensure that schools receive the 
support needed to implement policies, the SIG application criteria focused on practices for states 
and districts to use to give schools implementing SIG models (1) operational flexibility and 
(2) technical assistance and support. We identified two practices from the spring 2012 school 
administrator survey aligned with SIG objectives on operational flexibility and support 
(Table IV.4). 
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Table IV.4. Practices Aligned with SIG Objectives on Operational Flexibility and Support, by Subtopic 

Having Operational Flexibility 

School has primary responsibility for budget, hiring, discipline, or school year length decisions 

Receiving Technical Assistance and Support 

State, district, or an external support provider sponsored by the state or district provided training or technical 
assistance to support school improvement efforts or that the school received support to help administrators and 
teachers use data to improve instruction 

Source: SIG application; surveys of school administrators in spring 2012. 

Note: See Appendix D for a list of the specific survey questions that were aligned with the SIG practices in this 

table. 

In spring 2012, schools implementing a SIG-funded model reported using statistically 
significantly more of the practices aligned with the having operational flexibility and 
receiving support criteria in the SIG application than did schools not implementing a SIG- 
funded model. Schools implementing a SIG-funded model reported using an average of 1 .0 of 2 
practices in this area, compared to 0.8 for schools not implementing such a model (Figure IV. 9), 
a difference of 0.2 practices. 29 Therefore, although this difference was statistically significant, it 
may not be substantively important. 

In spring 2012, statistically significantly more schools implementing a SIG-funded 
model than schools not implementing such a model reported receiving technical assistance 
and support to help administrators and teachers use data to improve instruction. Fifteen 
percent of schools implementing a SIG-funded model reported receiving technical assistance and 
support, compared to 8 percent of schools not implementing such a model (Appendix A Figure 
A. 13). 

The individual practice with the highest level of usage in the operational flexibility and 
support area was receiving training or technical assistance from the state, district, or an 
external support provider sponsored by the state or district to support school improvement 
efforts or receiving support to help administrators and teachers use data to improve 
instruction. Across all study schools, the average value for this practice was 0.60. Average 
values for each group of schools (those implementing a SIG-funded intervention model and 
those not implementing one) are shown in Figure IV. 10. 

The individual practice with the lowest level of usage in the operational flexibility and 
support area was having primary responsibility for budget, hiring, discipline, or school 
year length decisions. Across all study schools, the average value for this practice was 0.30. 
Average values for each group of schools (those implementing a SIG-funded intervention model 
and those not implementing one) are shown in Figure IV. 10. 


29 These findings are not directly comparable to those presented in Herman et al. (2014) which examined 
individual survey questions (as opposed to the summary measures used in this report), focused on a somewhat 
different set of survey questions, and had less statistical power to detect differences (due to focusing on individual 
survey questions as opposed to a summary measure like the one used in this report). 
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Figure IV.9. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on Operational 
Flexibility and Support, Spring 2012 



Schools Implementing a SIG-Funded Schools Not Implementing a SIG-Funded 
Intervention Model in 2011-2012 Intervention Model in 2011-2012 

(N=290) (N=180) 


Source: Surveys of school administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table IV. 4. Each dot in this figure represents 

the number of schools that reported using a particular number of practices (out of two examined) that 
were aligned with the SIG application criteria. The number inside each dot is the number of schools 
represented by the dot; dots that represent less than 10 schools have no number inside. For example, 
37 schools implementing a SIG-funded intervention model reported using one of the two operational 
flexibility and support practices aligned with the SIG application criteria. For one practice, a “yes” 
response received one point. In the other case, it was possible for a school to receive a fraction of one 
point. See Chapter II for details on the way in which the number of practices was determined for each 
school. The dashed line denotes the average number of practices for each group of schools. 

"Significantly different from schools not implementing a SIG-funded intervention model at the 0.05 level, two-tailed 
test. 
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Figure IV.10. Study Schools’ Reported Usage of Individual Practices Aligned with SIG Objectives 
on Operational Flexibility and Support, Spring 2012 


Receiving Technical Assistance and Support 



Extent to Which School Has 
Autonomy on Budgeting, 
Hiring, Discipline, 
or School Year Length 


Extent to Which School 
Receives Training or 
Technical Assistance to 
Support School Improvement 
or Use Data to Improve Instruction 
Practice Aligned with SIG Objectives 


■ Schools Implementing a SIG-Funded Intervention Model in 2011-2012 (N=270-290) 
Schools Not Implementing a SIG-Funded Intervention Model in 2011-2012 (N=1 70— 1 80) 


Source: Surveys of school administrators in spring 2012. 

Note: This figure has a separate panel for each subtopic. As described in Chapter II, for each practice in the 

SIG application criteria for which we identified one or more survey questions aligned with the practice, 
we constructed a variable ranging from zero to one, with a value of one indicating that the school 
responded “yes” to all the survey questions selected for that practice. The height of each bar represents 
the mean value of the practice (on a scale of zero to one) for each group of schools. For some of the 
practices shown in this figure, multiple survey questions aligned with that practice. In the figure, we 
indicate this using the words “extent to which” at the beginning of the practice, to emphasize that the 
level of usage of that practice is measured using multiple survey questions (as opposed to a single, 
binary measure of whether that practice was used). A range is provided for the sample sizes because 
nonresponse varied across items. 


E. Summary 

This chapter examined the extent to which schools implementing a SIG-funded model and 
schools not implementing one reported using SIG-promoted practices in spring 2012 in four 
areas. As noted in Chapters II and III, the findings should be interpreted with caution because the 
SIG sample was not randomly selected and is not representative of schools nationwide. 

On average, schools implementing a SIG-funded model reported using statistically 
significantly more SIG-promoted practices than schools not implementing one in all four areas. 
The magnitude of these differences ranged from 0.2 to 1.5 practices per area (Figure IV. 11). 
Therefore, although these differences were statistically significant, they may not be substantively 
important. 
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Although the findings in this chapter focus on averages for the two groups of schools (those 
implementing a SIG-funded model and those not implementing one), it is important to keep in 
mind that there was substantial variation in the reported usage levels of SIG-promoted practices 
within each group. As a result, there was considerable overlap across groups in all four areas. In 
other words, many schools not implementing a SIG-funded model used more SIG-promoted 
practices than the average number used by schools implementing a SIG-funded model (Figures 
IV. 1, IV. 3, IV. 7, IV. 9); in the figures, these schools are the ones in the second column of (red) 
dots that are above the dashed average line that is shown for the first column of (blue) dots. In 
addition, many schools implementing a SIG-funded model used fewer SIG-promoted practices 
than the average number used by schools not implementing a SIG-funded model (Figures IV. 1, 
IV. 3, IV. 7, IV. 9); in the figures, these schools are the ones in the first column of (blue) dots that 
are below the dashed average line that is shown for the second column of (red) dots. 

Figure IV.11. Study Schools’ Reported Usage of Practices Promoted by SIG, by Topic Area 



Comprehensive 
Instructional Reform 
Strategies 
(8 items) 


Teacher and 
Principal 
Effectiveness 
(20 items) 


Learning Time and 
Community-Oriented 
Schools 
(5 items) 


Operational 
Flexibility 
and Support 
(2 items) 


■ Schools Implementing a SIG-Funded Intervention Model in 2011-2012 (N=290) 
Schools Not Implementing a SIG-Funded Intervention Model in 2011-2012 (N= 1 80) 


Source: Surveys of school administrators in spring 2012. 

Note: The total number of practices differs by topic area. This figure reads as follows (using the first bar on 

the left as an example): schools implementing a SIG-funded intervention model reported using 91 
percent of the practices in the comprehensive instructional reform strategies area, or 7.3 out of 8 
practices examined in that area. 

"Significantly different from schools not implementing a SIG-funded intervention model in 2011-2012 at the 0.05 
level, two-tailed test. 


Across all schools, usage of SIG-promoted practices was highest in the comprehensive 
instructional reform strategies area and lowest in the operational flexibility and support area. 
Study schools reported using, on average, 90 percent of the SIG-promoted practices in the 
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comprehensive instructional reform strategies area and 46 percent of the SIG-promoted practices 
in the operational flexibility and support area (not shown). 

Across all study schools, the individual practice with the highest level of usage was using 
benchmark or interim assessments at least once per year, with an average value of 0.99 (not 
shown). The individual practice with the lowest level of usage was using teacher evaluation 
results to inform decisions about compensation, with an average value of 0.15 (not shown). 
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V. EXTENT TO WHICH SCHOOLS FOCUS ON ENGLISH LANGUAGE LEARNERS 
IN THEIR USAGE OF PRACTICES PROMOTED BY SCHOOL 
IMPROVEMENT GRANTS 

English language learners (ELLs) are of particular interest to this evaluation because: 
(1) they are historically lower-achieving than non-ELLs, and (2) the SIG program placed 
particular emphasis on prioritizing the academic improvement of high-needs students, including 
ELLs. Since 2002, ELLs’ reading test scores have been below those of non-ELLs on the 
National Assessment of Educational Progress (NAEP). 30 The SIG program offered the 
opportunity to address this achievement gap. In particular, the SIG application criteria called 
upon districts and schools to provide supports and professional development to teachers and 
principals to ensure that ELLs acquire language skills to master academic content (U.S. 
Department of Education 2010). For these reasons, ED’s Office of English Language Acquisition 
(OELA) requested that part of this evaluation focus on how schools have addressed the needs of 
ELLs as they used the practices promoted by SIG. 

In this chapter, we assess the extent to which schools implementing a SIG-fimded 
intervention model and those not implementing one reported focusing on ELLs in their usage of 
practices promoted by SIG. 

We present results from four types of analyses: 

1. We compare usage of ELL-focused practices aligned with the SIG application criteria 
for schools implementing a SIG-funded model and schools not implementing one. 

2. We compare usage of these ELL-focused practices for schools implementing a SIG- 
funded model and schools not implementing one within each of the following four 
groups: 

a. schools with higher ELL populations (defined as schools with percentages of ELLs 
above the median percentage for our study sample). For example, within the group 
of schools with higher ELL populations, we compare schools implementing a SIG- 
funded model to schools not implementing one; 

b. schools with lower ELL populations (defined as schools with percentages of ELLs 
below the median percentage); 

c. schools with higher ELL/non-ELL achievement gaps (defined as schools with 
achievement gaps above the median gap for our study sample); and 

d. schools with lower ELL/non-ELL achievement gaps (defined as schools with 
achievement gaps below the median gap). 31 


30 National Center for Education Statistics. The Condition of Education. Accessed February 17, 2014 at 
https://nces.ed.gov/programs/coe/indicator_cgf.asp. 

31 We calculated the percentage of ELLs using student-level administrative data from 2009-2010, which 
contained indicators for whether each student participated in a program for ELLs. We calculated ELL/non-ELL 
achievement gaps as the average standardized score for non-ELLs minus the average standardized score for ELLs on 
the 2009-2010 state math assessment. See Chapter II for more details on how schools were classified into higher 
(above-median) and lower (below-median) groups. 
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3. Within each of the following two groups of schools — those implementing a SIG- 
funded model and those not implementing one — we compare usage of ELL-focused 
practices for schools that had higher and lower ELL populations. For example, within 
the group of schools implementing a SIG-funded model, we compare schools with 
higher ELL populations to schools with lower ELL populations. 

4. Within each of the following two groups of schools — those implementing a SIG- 
funded model and those not implementing one — we compare usage of ELL-focused 
practices for schools that had higher and lower ELL/non-ELL achievement gaps. 

Readers interested in specific examples of the individual ELL-focused practices included in 
these analyses may consult Appendix E. To provide context for school reports about ELL- 
focused practices aligned with the SIG application criteria, Appendix E also displays findings on 
the extent to which districts reported focusing on ELLs in their usage of practices promoted by 
SIG. As noted in Chapters II and III, the findings should be interpreted with caution because the 
SIG sample was not randomly selected and is not representative of schools or districts 
nationwide. 

Table V.l shows descriptive statistics on the distribution of the ELL population sizes and the 
ELL/non-ELL achievement gap for each group of schools (those implementing a SIG-funded 
model and those not implementing one). To calculate these variables, we first used student-level 
data to compute the percentage of students who were ELLs and the ELL/non-ELL achievement 
gap for each school in our SIG sample. We then used these school-level values to determine the 
median ELL population and median ELL/non-ELL achievement gap for the schools in our SIG 
sample. On average across all study schools, ELLs made up 17.7 percent of the student body, 
and non-ELLs performed 0.31 standard deviations higher than ELLs on the state math 
assessment. Schools implementing a SIG-funded model had a lower average ELL/non-ELL 
achievement gap than schools not implementing one (0.26 versus 0.38 percent). As a result, a 
smaller proportion of schools implementing a SIG-funded model (45.5 percent) were classified 
into the group of schools with higher ELL/non-ELL achievement gaps than schools not 
implementing such a model (57.6 percent). 

Table V.l . Distribution of ELL Population and ELL/Non-ELL Achievement Gap 

Study Schools Not 

Study Schools Implementing Implementing a SIG-Funded 
a SIG-Funded Intervention Intervention Model in 

All Study Schools Model in 201 1-2012 2011-2012 


Distribution of ELL Population 


10th Percentile 

0.5 

0.3 

1.1 

50th Percentile 

13.3 

13.6 

12.9 

90th Percentile 

42.7 

44.7 

35.4 

Mean 

17.7 

18.3 

16.7 

Percentage of Schools That Had Higher and Lower ELL Populations 

Higher 

50.0 

50.6 

49.0 

Lower 

50.0 

49.4 

51.0 

Distribution of ELL/Non-ELL Achievement Gap 

10th Percentile 3 

-0.22 

-0.25* 

-0.01 

50th Percentile 

0.33 

0.28* 

0.39 

90th Percentile 

0.78 

0.79* 

0.78 
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Table V.1 (continued) 



All Study Schools 

Study Schools Implementing 
a SIG-Funded Intervention 
Model in 2011-2012 

Study Schools Not 
Implementing a SIG-Funded 
Intervention Model in 
2011-2012 

Mean 

0.31 

0.26* 

0.38 


Percentage of Schools That Had Higher and Lower ELL/Non-ELL Achievement Gaps 

Higher 

50.1 

45.5* 

57.6 

Lower 

49.9 

54.5* 

42.4 

Number of Schools 360-400 

220-250 

140-150 

Source: 

Note: 

State and district administrative records. 
We calculated the ELL population using 

student-level administrative 

data from 2009-2010, which 


contained indicators for whether each student participated in a program for ELLs. We calculated 
ELL/non-ELL achievement gaps as the average standardized score for non-ELLs minus the average 
standardized score for ELLs on the 2009-2010 state math assessment. All scores were standardized to 
have a standard deviation of 1, so results are reported in effect size units. Schools were classified into 
higher and lower groups based on whether their value (for either the ELL population or the ELL/non-ELL 
achievement gap) was above or below the median value across all study schools. See Chapter II for 
more details on how schools were classified into groups. Schools that had no ELLs were not included in 
the analysis. A range is provided for the sample size because missing data varied across items. 
a The negative numbers in this row indicate that there were some schools in which ELLs scored higher than non- 
ELLs on average. 

‘Significantly different from schools not implementing a SIG-funded intervention model in 2011-2012 at the 0.05 
level, two-tailed test. 

ELL = English language learner. 

We identified six ELL-focused practices from the spring 2012 school administrator survey 
aligned with SIG objectives (Table V.2). 

Table V.2. ELL-Focused Practices Aligned with SIG Objectives 

Teachers have the opportunity to receive financial incentives designed to increase the number of staff 

with ELL expertise 

Principals have the opportunity to receive financial incentives designed to increase the number of staff 

with ELL expertise 

Using data on ELLs to inform and differentiate instruction 

Implementing strategies (including additional supports or PD) to ensure that limited English proficient 

students acquire language skills to master academic content 

Providing additional services for ELLs (such as tutors, bilingual aides, or an after-school program) 

Receiving supports from the state education agency or local education agency to use data on ELLs to 
improve or differentiate instruction 

Source: SIG application; surveys of school administrators in spring 2012. 

Note: See Appendix E for a list of the specific survey questions that were aligned with the ELL- 

focused practices in this table. All the practices listed in this table were included in the main 
analyses described in Chapter IV, but some of them are not listed in the Chapter IV tables 
because they were included in a broader practice that is listed in those tables. 

ELL = English language learner. 

In spring 2012, there was no statistically significant difference in usage of ELL-focused 
practices aligned with the SIG application criteria between schools implementing a SIG- 
funded model and schools not implementing one. Schools implementing a SIG-funded model 
reported using an average of 3.1 of 6 ELL-focused practices aligned with the SIG application 
criteria, compared to 3.0 for schools not implementing such a model (Figure V.l). 
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Figure V.l. Schools’ Reported Usage of ELL-Focused Practices Aligned with SIG Objectives, Spring 2012 



Schools Implementing a SIG-Funded Schools Not Implementing a SIG-Funded 
Intervention Model in 2011-2012 Intervention Model in 2011-2012 

(N=230) (N=140) 


Source: Surveys of school administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table V.2. Each dot in this figure represents the number of 

schools that reported using a particular number of ELL-focused practices (out of six examined) that were aligned with 
the SIG application criteria. The number inside each dot is the number of schools represented by the dot; dots that 
represent less than 10 schools have no number inside. For example, 21 schools implementing a SIG-funded 
intervention model reported using four of six ELL-focused practices aligned with the SIG application criteria. For 
three of the ELL-focused practices, a “yes" response received one point. In the other three cases, it was possible for 
a school to receive a fraction of one point. See Chapter II for details on the way in which the number of ELL-focused 
practices was determined for each school. The dashed line denotes the average number of ELL-focused practices 
for each group of schools. Schools that had no ELLs were not included in the analysis. The sample sizes in this 
figure are smaller than those in Table V.l because some schools had a missing value for all six ELL-focused 
practices and were therefore excluded from the analysis for this figure. There were no statistically significant 
differences between schools implementing a SIG-funded intervention model in 2011-2012 and schools not 
implementing one at the 0.05 level using a two-tailed test. 
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Among schools with higher ELL populations, schools implementing a SIG-funded 
intervention model reported using statistically significantly more ELL-focused practices 
aligned with the SIG application criteria than schools not implementing one. Among 
schools with higher ELL populations, schools implementing a SIG-funded intervention model 
reported using 3.6 of 6 ELL-focused practices, compared with 3.3 for schools not implementing 
such a model (Ligure V.2). Among schools with lower ELL populations and lower or higher 
ELL/non-ELL achievement gaps, there were no statistically significant differences in usage of 
ELL-focused practices aligned with the SIG application criteria between schools implementing a 
SIG-funded intervention model and schools not implementing one. 

Within each of the following two groups — schools implementing a SIG-funded model 
and schools not implementing one — schools with higher ELL populations reported using 
statistically significantly more ELL-focused practices than schools with lower ELL 
populations. Among schools implementing a SIG-funded model, schools with higher ELL 
populations reported using 3.6 of 6 ELL-focused practices aligned with the SIG application 
criteria, compared with 2.4 practices for schools with lower ELL populations (Ligure V.2). 
Among schools not implementing a SIG-funded model, schools with higher ELL populations 
reported using 3.3 of 6 ELL-focused practices aligned with the SIG application criteria, 
compared with 2.7 practices for schools with lower ELL populations. 

Within each of the following two groups — schools implementing a SIG-funded model 
and schools not implementing one — there was no statistically significant difference in usage 
of the ELL-focused practices aligned with the SIG application criteria between schools with 
higher and lower ELL/non-ELL achievement gaps. Among schools implementing a SIG- 
funded model, schools with higher ELL/non-ELL achievement gaps reported using 3.3 of 6 
practices aligned with SIG application criteria, compared with 3.0 practices for schools with 
lower ELL/non-ELL achievement gaps (Ligure V.3). Among schools not implementing a SIG- 
funded model, schools with higher achievement gaps reported using 3.2 of 6 practices, compared 
with 2.9 practices for schools with lower achievement gaps. 

The individual ELL-focused practice with the highest level of usage was using data on 
ELLs to inform and differentiate instruction. Across all study schools, the average value for 
this practice was 0.83. Average values for each group of schools (those implementing a SIG- 
funded intervention model and those not implementing one) are shown in Ligure V.4. 

The individual ELL-focused practice with the lowest level of usage was principals 
having the opportunity to receive financial incentives designed to increase the number of 
staff with ELL expertise. Across all study schools, the average value for this practice was 0.03. 
Average values for each group of schools (those implementing a SIG-funded intervention model 
and those not implementing one) are shown in Ligure V.4. 
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Figure V.2. Schools’ Reported Usage of ELL-Focused Practices Aligned with SIG Objectives, by ELL 
Population, Spring 2012 


Schools Implementing a 
SIG-Funded Intervention Model 
in 2011-2012 
(N=230) 


Schools Not Implementing a 
SIG-Funded Intervention Model 
in 2011-2012 
(N=1 40) 
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Source: Surveys of school administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table V.2. Each column in the figure shows the number of 

ELL-focused practices that schools in each group reported using, by schools that had higher and lower ELL 
populations. Each dot in this figure represents the number of schools that reported using a particular number of ELL- 
focused practices (out of six examined) that were aligned with the SIG application criteria. The number inside each 
dot is the number of schools represented by the dot; dots that represent less than 10 schools have no number inside. 
For example, 17 schools implementing a SIG-funded intervention model that had a higher ELL population reported 
using four of six ELL-focused practices aligned with the SIG application criteria. For three of the practices, a “yes” 
response received one point. In the other three cases, it was possible for a school to receive a fraction of one point. 
See Chapter II for details on the way in which the number of ELL-focused practices was determined for each school. 
The dashed line denotes the average number of ELL-focused practices for each group of schools. Schools that had 
no ELLs were not included in the analysis. The sample sizes in this figure are smaller than those in Table V.1 
because some schools had a missing value for all six ELL-focused practices and were therefore excluded from the 
analysis for this figure. 

•Significantly different at the 0.05 level, two-tailed test, from schools in the same ELL subgroup not implementing a SIG-funded 
intervention model. For example, schools implementing a SIG-funded intervention model with higher ELL populations reported using 
statistically significantly more ELL-focused practices than schools not implementing a SIG-funded intervention model with higher 
ELL populations. 

^Significantly different at the 0.05 level, two-tailed test, from schools with the same SIG-funded intervention model status but lower 
ELL populations. For example, among schools implementing a SIG-funded intervention model, schools with higher ELL populations 
reported using statistically significantly more ELL-focused practices than schools with lower ELL populations. 
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Figure V.3. Schools’ Reported Usage of ELL-Focused Practices Aligned with SIG Objectives, by ELL/Non-ELL 
Achievement Gap, Spring 2012 


Schools Implementing a 
SIG-Funded Intervention Model 
in 2011-2012 
(N=210) 


Schools Not Implementing a 
SIG-Funded Intervention Model 
in 2011-2012 
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Source: Surveys of school administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table V.2. Each column in the figure shows the 

number of ELL-focused practices that schools in each group reported using, by schools that had higher and 
lower achievement gaps between ELL and non-ELLs. Each dot in this figure represents the number of schools 
that reported using a particular number of ELL-focused practices (out of six examined) that were aligned with 
the SIG application criteria. The number inside each dot is the number of schools represented by the dot; dots 
that represent less than 10 schools have no number inside. For example, 13 schools implementing a SIG- 
funded intervention model that had a lower ELL/non-ELL achievement gap reported using just under four of 
six ELL-focused practices aligned with the SIG application criteria. For three of the ELL-focused practices, a 
“yes” response received one point. In the other three cases, it was possible for a school to receive a fraction 
of one point. See Chapter II for details on the way in which the number of ELL-focused practices was 
determined for each school. The dashed line denotes the average number of ELL-focused practices for each 
group of schools. Schools that had no ELLs were not included in the analysis. The sample sizes in this figure 
are smaller than those in Figures V.1 and V.2 because some schools with low percentages of ELLs did not 
have test score data available for ELLs, so the ELL/non-ELL achievement gap could not be calculated. There 
were no statistically significant differences at the 0.05 level using a two-tailed test (1) between schools 
implementing and not implementing a SIG-funded intervention model with the same ELL/non-ELL 
achievement gap classification, and (2) between higher and lower ELL/non-ELL achievement gap schools 
with the same SIG-funded intervention model implementation status. 
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Figure V.4. Schools’ Reported Usage of Individual ELL-Focused Practices Aligned with SIG 
Objectives, Spring 2012 



Provides Financial 
Incentives to 
Increase Teachers 
with ELL 
Expertise 


Provides Financial 
Incentives to 
Increase Principals 
with ELL 
Expertise 


Extent to 
Which School 
Uses Data on 
ELLs to Inform 
and Differentiate 
Instruction 


Extent to 
Which School 
Implements 
Strategies to 
Meet Needs 
of ELLs 


Provides Additional 
Services for ELLs 


Extent to 
Which School 
Receives Supports 
to Use Data on 
ELLs For 
Instruction 


ELL-Focused Practice Aligned with SIG Objectives 


■ Schools Implementing a SIG-Funded Intervention Model in 2011-2012 (N=160-210) 
Schools Not Implementing a SIG-Funded Intervention Model in 2011-2012 (N=1 20— 1 40) 


Source: Surveys of school administrators in spring 2012. 

Note: As described in Chapter II, for each ELL-focused practice aligned with the SIG application 

criteria for which we identified one or more survey questions that addressed the practice, we 
constructed a variable ranging from zero to one, with a value of one indicating that the school 
responded “yes” to all the survey questions selected for that practice. The height of each bar 
represents the mean value of the ELL-focused practice (on a scale of zero to one) for each 
group of schools. For some of the practices shown in this figure, multiple survey questions 
aligned with that practice. In the figure, we indicate this using the words “extent to which” at 
the beginning of the practice, to emphasize that the level of usage of that practice is 
measured using multiple survey questions (as opposed to a single, binary measure of 
whether that practice was used). 

ELL = English language learner. 
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VI. DISCUSSION OF MAIN FINDINGS FOR SCHOOL IMPROVEMENT GRANTS 

As noted in Chapter I, few studies on the implementation of SIG-promoted practices 
examine whether the practices used by SIG schools differ from those used by schools that did not 
receive SIG. This volume of the report seeks to address this gap in the existing literature by 
examining the extent to which school administrators in schools that did and did not implement a 
SIG-funded intervention model reported that their schools used SIG-promoted practices in four 
topic areas: (1) adopting comprehensive instructional refonn strategies, (2) developing and 
increasing teacher and principal effectiveness, (3) increasing learning time and creating 
community-oriented schools, and (4) having operational flexibility and receiving support. We 
also assessed the extent to which schools implementing a SIG-funded model and those not 
implementing one reported focusing on ELLs in their usage of practices promoted by SIG. 

As noted in Chapters II and III, the findings should be interpreted with caution because the 
SIG sample was not randomly selected and is not representative of schools nationwide. In 
interpreting the results, please also note the following caveats: (1) the findings are based on self- 
reported usage of practices, (2) our study instruments did not address every practice listed in the 
SIG application, (3) the application wording left it up to the schools to decide many of the details 
about how to implement particular practices, and (4) we did not collect information about the 
quality or fidelity with which the practices were implemented. 

Schools implementing a SIG-funded model reported using statistically significantly more of 
the practices promoted by SIG, on average, than schools not implementing a SIG-funded model 
in all four topic areas considered and for 7 of the 13 subtopics covered under these four topic 
areas. This finding is consistent with a prior study that showed SIG schools reported 
implementing school improvement practices more intensively than non-SIG schools (Center on 
Education Policy 2012a). Although our finding focuses on averages for each group of schools, 
there was substantial variation in the reported usage of SIG-promoted practices within each 
group. In addition, although we found that the differences between schools implementing a SIG- 
funded model and schools not implementing one were statistically significant in all four areas, 
they may not be substantively important, given that the difference between the two groups in the 
average number of SIG-promoted practices used ranged from 0.2 to 1.5 practices per area. 

Usage of practices promoted by SIG was highest in the comprehensive instructional reform 
strategies area — which is consistent with findings from an earlier study (U.S. Department of 
Education 2007) — and lowest in the operational flexibility and support area. Study schools 
reported using, on average, 90 percent of the SIG-promoted practices in the comprehensive 
instructional reform strategies area and 46 percent of the SIG-promoted practices in the 
operational flexibility and support area. Levels of usage for the other areas — developing and 
increasing teacher and principal effectiveness, and increasing learning time and creating 
community-oriented schools — were 53 and 73 percent. 

Across all study schools, the individual practice with the highest level of usage was using 
benchmark or interim assessments at least once per year. The individual practice with the lowest 
level of usage was using teacher evaluation results to inform decisions about compensation. 

We found no statistically significant differences in usage of ELL-focused practices 
promoted by SIG between schools implementing a SIG-funded model and schools not 
implementing one (both groups of schools reported using an average of 2.9 practices out of 6 
examined). Within each group of schools (those implementing a SIG-funded model and those not 
implementing one), schools with higher ELL populations used statistically significantly more 
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ELL-focused practices than schools with lower ELL populations, but there were no statistically 
significant differences in usage between schools with higher and lower ELL/non-ELL 
achievement gaps. 

Readers may have questions about potential explanations for this pattern of findings. Below, 
we lay out several questions of possible interest and potential explanations for these findings. 

Why did schools implementing a SIG-funded model report using more practices 
aligned with the SIG application than schools not implementing a SIG-funded model? One 

potential explanation is that the SIG program had an effect on schools’ usage of these practices. 
The practices examined in this report were required or permissible activities under the SIG 
transformation or turnaround models, and most of the study schools that were implementing a 
SIG-funded model reported using one of these models (46 percent for the transfonnation model 
and 44 percent for the turnaround). Therefore, the observed differences could be due to those 
schools adhering to the grant requirements or using practices permissible by SIG, while schools 
not implementing a SIG-funded model were not subject to those same obligations. Another 
possible explanation is that this finding could reflect differences in the usage of these practices 
between the groups of schools prior to the SIG awards. To limit the length of the school survey, 
we did not include questions about practices used prior to SIG awards, so we could not 
detennine whether differences in usage of these practices existed at baseline. As noted earlier, 
although the differences between schools implementing a SIG-funded model and those not doing 
so were statistically significant, they were not necessarily substantively important. For example, 
the difference was 0.3 practices for the comprehensive instructional reform strategies area and 
1 .5 practices for the teacher and principal effectiveness area. 

Why did schools implementing a SIG-funded model not report using statistically 
significantly more SIG practices than schools not implementing such a model for some of 
the examined subtopics? For 6 of 13 subtopics, responses from schools implementing a SIG- 
funded model and from those not implementing such a model did not differ by a statistically 
significant margin. One possible explanation for these results is that schools implementing a 
SIG-funded model encountered barriers in implementing some practices in these specific areas. 
For example, schools may have encountered challenges with implementing new data systems or 
working with teachers within the requirements of collective bargaining agreements — to name 
two examples of challenges related to subtopics where no statistically significant differences 
emerged. This hypothesis is consistent with prior literature that showed common challenges 
encountered during SIG implementation included difficulties using data to inform and 
differentiate instruction and limited state capacity to provide assistance (Center on Education 
Policy 2012a; GAO 2011; U.S. Department of Education 201 la-j). Another possibility is that the 
practices examined in these areas had also been used by schools not implementing a SIG-funded 
model, perhaps because of district requirements, outside resources that allowed schools to use 
these practices, or enhanced awareness of the practices as a result of SIG or other programs that 
also promoted them. A third possibility is that some of the practices examined within some of 
these subtopics were permissible under SIG rather than required, so that schools implementing 
SIG-funded models could choose not to use them. In fact, for 2 of the 6 subtopics for which no 
statistically significant differences emerged, all of the examined practices were pennissible 
rather than required, and for one subtopic, one-third of the examined practices were pennissible 
(for the remaining three subtopics, all of the examined practices were required). In addition, 
some of the practices examined were not required for schools implementing the restart or closure 
models, but since these schools make up a small proportion of study schools (less than 10 
percent), it is unlikely that they have a big effect on the overall results. 
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Why did schools implementing a SIG-funded model not report using more of the SIG 
practices? Schools implementing a SIG-funded model reported using about half of the practices 
on developing and increasing teacher and principal effectiveness, and having operational 
flexibility and receiving support. One possible explanation for these results is that schools are 
still in the process of adopting these practices, which take time to implement. Although schools 
were in the second year of their three-year grants at the time of our spring 2012 interviews, 
recent studies suggested that some schools experienced delays in implementing a SIG-funded 
model for a variety of reasons, including late federal approval of the state’s SIG application 
(Center on Education Policy 2012a and 2012b; GAO 2011; U.S. Department of Education 
20 1 1 a— j). Another possibility is that schools chose to focus their efforts on a select group of 
practices in each area rather than using all of them, perhaps because they had technical capacity 
constraints or because they perceived certain practices as more important than others to their 
particular reform agenda and unique needs. 

Why did schools implementing a SIG-funded model not report using more ELL- 
focused practices promoted by SIG than schools not implementing a SIG-funded model? 

We focus here on two potential explanations for this finding. First, it is possible that SIG did not 
have an effect on ELL-focused practices among schools. This could be because schools are 
focused on practices thought to be effective for all students, including ELLs, rather than practices 
explicitly focused on ELLs. Second, it is possible that ELL-focused practices promoted by SIG 
are on the rise primarily in schools with higher ELL populations (which include some schools 
implementing a SIG-funded model and some schools not implementing one). Our finding that 
within each of those two groups of schools, schools with higher ELL populations reported using 
statistically significantly more ELL-focused practices than schools with lower ELL populations 
is consistent with this hypothesis. 

Although we cannot definitively accept or reject any of these possible explanations for these 
findings, we offer them as starting points for future investigations into the implementation of 
SIG school improvement practices. Because the process of changing education practices can be 
complex and require substantial time to implement, we administered a second survey of school 
administrators in spring 2013 so that we can continue to explore schools’ progress toward usage 
of SIG practices after another year of the program. We will revisit these patterns and findings in 
a future report. 
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This appendix contains additional figures that are directly related to the analyses presented 
in Chapter IV. In this appendix, we focus on the same four topic areas addressed in Chapter IV : 
(1) adopting comprehensive instructional reform strategies, (2) developing and increasing 
teacher and principal effectiveness, (3) increasing learning time and creating community- 
oriented schools, and (4) having operational flexibility and receiving support. For each area, we 
present a series of figures, one for each subtopic, showing schools’ reported usage of the 
practices aligned with the School Improvement Grants (SIG) application criteria for that 
subtopic, similar to the figures shown in Chapter IV for each topic area. 
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A. Comprehensive Instructional Reform Strategies 

Figure A.1. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on 
Comprehensive Instructional Reform Strategies, Using Data to Identify and Implement an 
Instructional Program Subtopic, Spring 2012 
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Schools Implementing a SIG-Funded 
Intervention Model in 2011-2012 
(N=290) 


Schools Not Implementing a SIG-Funded 
Intervention Model in 2011-2012 
(N=170) 


Source: Surveys of school administrators in spring 2012. 

Note: The practice summarized in this figure is presented in Table IV.1. Each dot in this figure 

represents the schools that reported using the one practice that was aligned with the SIG 
application criteria. The number inside each dot is the number of schools represented by the 
dot. To protect respondent confidentiality, the number inside the smallest dot for each group 
of schools has been removed. For this practice, a “yes” response received one point. See 
Chapter II for details on the way in which the number of practices was determined for each 
school. The dashed line denotes the average number of practices for each group of schools. 
There were no statistically significant differences between schools implementing a SIG- 
funded intervention model in 2011-2012 and schools not implementing one at the 0.05 level 
using a two-tailed test. 
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Figure A.2. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on 
Comprehensive Instructional Reform Strategies, Promoting the Continuous Use of Student Data 
Subtopic, Spring 2012 



(N=290) (N=180) 

Source: Surveys of school administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table IV. 1. Each dot in this figure 

represents the schools that reported using a particular number of practices (out of two 
examined) that were aligned with the SIG application criteria. The number inside each dot is 
the number of schools represented by the dot; dots that represent less than 10 schools have 
no number inside. For both practices, it was possible for a school to receive a fraction of one 
point. See Chapter II for details on the way in which the number of practices was determined 
for each school. The dashed line denotes the average number of practices for each group of 
schools. 

*Significantly different from schools not implementing a SIG-funded intervention model at the 0.05 level, 
two-tailed test. 
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Figure A.3. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on 
Comprehensive Instructional Reform Strategies, Providing Supports and Professional 
Development to Staff to Assist English Language Learners and Students with Disabilities 
Subtopic, Spring 2012 



(N=220) (N=150) 

Source: Surveys of school administrators in spring 2012. 

Note: The practice summarized in this figure is presented in Table IV. 1. This figure presents one practice 

described in the SIG application criteria to which multiple survey questions aligned. As described in 
Chapter II, whenever multiple survey questions aligned with a single practice from the application 
criteria, we used those questions to construct a variable ranging from zero to one, with schools 
receiving a fraction of a point for each question to which they responded “yes.” Each dot in this figure 
represents the schools that reported using a particular proportion of the survey questions aligned to the 
practice described in the SIG application criteria. The number inside each dot is the number of schools 
represented by the dot; dots that represent less than 10 schools have no number inside. The dashed 
line denotes the average value for each group of schools. There were no statistically significant 
differences between schools implementing a SIG-funded intervention model in 2011-2012 and schools 
not implementing one at the 0.05 level using a two-tailed test. 

ELLs = English language learners; PD = professional development. 
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Figure A.4. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on 
Comprehensive Instructional Reform Strategies, Using and Integrating Technology-Based 
Supports Subtopic, Spring 2012 



(N=280) (N=170) 

Source: Surveys of school administrators in spring 2012. 

Note: The practice summarized in this figure is presented in Table IV. 1. This figure presents one practice 

described in the SIG application criteria to which multiple survey questions aligned. As described in 
Chapter II, whenever multiple survey questions aligned with a single practice from the application 
criteria, we used those questions to construct a variable ranging from zero to one, with schools 
receiving a fraction of a point for each question to which they responded “yes.” Each dot in this figure 
represents the schools that reported using a particular proportion of the survey questions aligned to the 
practice described in the SIG application criteria. The number inside each dot is the number of schools 
represented by the dot; dots that represent less than 10 schools have no number inside. The dashed 
line denotes the average value for each group of schools. 

"Significantly different from schools not implementing a SIG-funded intervention model at the 0.05 level, two-tailed 
test. 
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Figure A.5. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on 
Comprehensive Instructional Reform Strategies, Tailoring Strategies for Secondary Schools 
Subtopic, Spring 2012 



(N=1 40) (N=80) 

Source: Surveys of school administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table IV. 1. Each dot in this figure 

represents the schools that reported using a particular number of practices (out of three 
examined) that were aligned with the SIG application criteria. The number inside each dot is 
the number of schools represented by the dot; dots that represent less than 10 schools have 
no number inside. For one practice, a “yes” response received one point. In the other two 
cases, it was possible for a school to receive a fraction of one point. See Chapter II for details 
on the way in which the number of practices was determined for each school. The dashed 
line denotes the average number of practices for each group of schools. There were no 
statistically significant differences between schools implementing a SIG-funded intervention 
model in 2011-2012 and schools not implementing one at the 0.05 level using a two-tailed 
test. 


A. 7 


Appendix A: Additional Figures Based on School Surveys 


Usage of Practices Promoted by SIG 


B. Teacher and Principal Effectiveness 

Figure A.6. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on Teacher 
and Principal Effectiveness, Using Rigorous, Transparent, and Equitable Evaluation Systems 
Subtopic, Spring 2012 
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Schools Implementing a SIG-Funded 
Intervention Model in 2011-2012 
(N=280) 


Schools Not Implementing a SIG-Funded 
Intervention Model in 2011-2012 
(N=170) 


Source: Surveys of school administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table IV.2. Each dot in this figure 

represents the schools that reported using a particular number of practices (out of four 
examined) that were aligned with the SIG application criteria. The number inside each dot is 
the number of schools represented by the dot; dots that represent less than 10 schools have 
no number inside. For one practice, a “yes” response received one point. In the other three 
cases, it was possible for a school to receive a fraction of one point. See Chapter II for details 
on the way in which the number of practices was determined for each school. The dashed 
line denotes the average number of practices for each group of schools. There were no 
statistically significant differences between schools implementing a SIG-funded intervention 
model in 2011-2012 and schools not implementing one at the 0.05 level using a two-tailed 
test. 
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Figure A.7. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on Teacher 
and Principal Effectiveness, Identifying and Rewarding Effective Teachers and Principals and 
Removing Ineffective Ones Subtopic, Spring 2012 



(N=280) (N=170) 

Source: Surveys of school administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table IV.2. Each dot in this figure 

represents the schools that reported using a particular number of practices (out of four 
examined) that were aligned with the SIG application criteria. The number inside each dot is 
the number of schools represented by the dot; dots that represent less than 10 schools have 
no number inside. For all four practices, it was possible for a school to receive a fraction of 
one point. See Chapter II for details on the way in which the number of practices was 
determined for each school. The dashed line denotes the average number of practices for 
each group of schools. 

*Significantly different from schools not implementing a SIG-funded intervention model at the 0.05 level, 
two-tailed test. 
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Figure A.8. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on Teacher 
and Principal Effectiveness, Providing High Quality, Job-Embedded Professional Development or 
Supports Subtopic, Spring 2012 



0 - 


Schools Implementing a SIG-Funded Schools Not Implementing a SIG-Funded 
Intervention Model in 2011-2012 Intervention Model in 2011-2012 

(N=290) (N=180) 

Source: Surveys of school administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table IV. 2. Each dot in this figure represents 

the schools that reported using a particular number of practices (out of nine examined) that were 
aligned with the SIG application criteria. The number inside each dot is the number of schools 
represented by the dot; dots that represent less than 1 0 schools have no number inside. For four of the 
practices, a “yes” response received one point. In the other five cases, it was possible for a school to 
receive a fraction of one point. See Chapter II for details on the way in which the number of practices 
was determined for each school. The dashed line denotes the average number of practices for each 
group of schools. 

PD = professional development. 

‘Significantly different from schools not implementing a SIG-funded intervention model at the 0.05 level, two-tailed 

test. 
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Figure A.9. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on Teacher 
and Principal Effectiveness, Implementing Strategies to Recruit, Place, and Retain Staff Subtopic, 
Spring 2012 



Schools Implementing a SIG-Funded Schools Not Implementing a SIG-Funded 
Intervention Model in 2011-2012 Intervention Model in 2011-2012 

(N=290) (N=170) 


Source: Surveys of school administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table IV.2. Each dot in this figure 

represents the schools that reported using a particular number of practices (out of three 
examined) that were aligned with the SIG application criteria. The number inside each dot is 
the number of schools represented by the dot; dots that represent less than 10 schools have 
no number inside. For all three practices, it was possible for a school to receive a fraction of 
one point. See Chapter II for details on the way in which the number of practices was 
determined for each school. The dashed line denotes the average number of practices for 
each group of schools. There were no statistically significant differences between schools 
implementing a SIG-funded intervention model in 2011-2012 and schools not implementing 
one at the 0.05 level using a two-tailed test. 
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C. Learning Time and Community-Oriented Schools 

Figure A.10. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on Learning 
Time and Community-Oriented Schools, Increasing Learning Time Subtopic, Spring 2012 



(N=280) (N=170) 

Surveys of school administrators in spring 2012. 

The practice summarized in this figure is presented in Table IV. 3. This figure presents one practice 
described in the SIG application criteria to which multiple survey questions aligned. As described in 
Chapter II, whenever multiple survey questions aligned with a single practice from the application 
criteria, we used those questions to construct a variable ranging from zero to one, with schools 
receiving a fraction of a point for each question to which they responded “yes.” Each dot in this figure 
represents the schools that reported using a particular proportion of the survey questions aligned to the 
practice described in the SIG application criteria. The number inside each dot is the number of schools 
represented by the dot; dots that represent less than 10 schools have no number inside. The dashed 
line denotes the average value for each group of schools. 

‘Significantly different from schools not implementing a SIG-funded intervention model at the 0.05 level, two-tailed 
test. 


Source: 

Note: 
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Figure A.11. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on Learning 
Time and Community-Oriented Schools, Engaging Families and Communities and Providing a 
Safe School Environment that Meets Students’ Social, Emotional, and Health Needs Subtopic, 
Spring 2012 



(N=290) (ISM 80) 

Source: Surveys of school administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table IV. 3. Each dot in this figure represents 

the schools that reported using a particular number of practices (out of four examined) that were aligned 
with the SIG application criteria. The number inside each dot is the number of schools represented by 
the dot; dots that represent less than 10 schools have no number inside. For two of the practices, a 
“yes” response received one point. In the other cases, it was possible for a school to receive a fraction 
of one point. See Chapter II for details on the way in which the number of practices was determined for 
each school. The dashed line denotes the average number of practices for each group of schools. 

‘Significantly different from schools not implementing a SIG-funded intervention model at the 0.05 level, two-tailed 
test. 
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D. Operational Flexibility and Support 

Figure A.12. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on 
Operational Flexibility and Support, Having Operational Flexibility Subtopic, Spring 2012 



(N=270) (N=170) 

Source: Surveys of school administrators in spring 2012. 

Note: The practice summarized in this figure is presented in Table IV.4. This figure presents one 

practice described in the SIG application criteria to which multiple survey questions aligned. 
As described in Chapter II, whenever multiple survey questions aligned with a single practice 
from the application criteria, we used those questions to construct a variable ranging from 
zero to one, with schools receiving a fraction of a point for each question to which they 
responded “yes.” Each dot in this figure represents the schools that reported using a 
particular proportion of the survey questions aligned to the practice described in the SIG 
application criteria. The number inside each dot is the number of schools represented by the 
dot; dots that represent less than 10 schools have no number inside. The dashed line 
denotes the average value for each group of schools. There were no statistically significant 
differences between schools implementing a SIG-funded intervention model in 2011-2012 
and schools not implementing one at the 0.05 level using a two-tailed test. 
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Figure A.13. Study Schools’ Reported Usage of Practices Aligned with SIG Objectives on 
Providing Operational Flexibility and Support, Receiving Technical Assistance and Support 
Subtopic, Spring 2012 



(N=290) (ISM 80) 

Source: Surveys of school administrators in spring 2012. 

Note: The practice summarized in this figure is presented in Table IV.4. This figure presents one practice 

described in the SIG application criteria to which multiple survey questions aligned. As described in 
Chapter II, whenever multiple survey questions aligned with a single practice from the application 
criteria, we used those questions to construct a variable ranging from zero to one, with schools 
receiving a fraction of a point for each question to which they responded “yes.” Each dot in this figure 
represents the schools that reported using a particular proportion of the survey questions aligned to the 
practice described in the SIG application criteria. The number inside each dot is the number of schools 
represented by the dot; dots that represent less than 10 schools have no number inside. The dashed 
line denotes the average value for each group of schools. 

‘Significantly different from schools not implementing a SIG-funded intervention model at the 0.05 level, two-tailed 
test. 
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Appendix B: District-Reported Practices Aligned with the SIG Application Criteria 


Usage of Practices Promoted by SIG 


In contrast to the main body of the report and Appendix A, which summarized the extent to 
which schools reported using practices promoted by a School Improvement Grant (SIG), this 
appendix summarizes the extent to which district administrators reported using the practices 
promoted by SIG in spring 2012. The overarching research question answered by these district 
findings is: How are districts supporting schools ’ efforts to use practices promoted by SIG? For 
example, some of the school survey questions asked schools if they received particular types of 
support from districts or states. The findings in this appendix shed light on the extent to which 
districts reported providing those types of support. 

In this appendix, we focus on the same four topic areas addressed in Chapter IV: 
(1) adopting comprehensive instructional reform strategies, (2) developing and increasing 
teacher and principal effectiveness, (3) increasing learning time and creating community- 
oriented schools, and (4) having operational flexibility and receiving support. For each area, we 
first present a table that shows the practices from the district interview that aligned with the SIG 
application criteria. We then present a series of figures that display the results. The first figure 
displays the results of the overall analysis for the area. The figures that follow display the results 
for each subtopic within that topic area. 

The data presented in this appendix came from structured telephone interviews with 
administrators in the 60 districts where the SIG-sample schools were located. The interviews, 
conducted in spring 2012, documented the school turnaround practices being used and addressed 
both state- and district-level supports for those practices. 

One important difference between the figures shown in Chapter IV and the figures shown in 
this appendix is that the latter have no comparison group. All districts in the study sample 
included schools that were and were not implementing a SIG-funded intervention model. 
Therefore, in this appendix, we are not presenting comparisons between districts; instead, we are 
presenting descriptive information about the practices that study districts reported using. 

A. Comprehensive instructional Reform Strategies 

We identified seven practices from the spring 2012 district interview aligned with SIG 
objectives on comprehensive instructional refonn strategies (see Table B.l). 
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Table B.l. Practices Aligned with SIG Objectives on Comprehensive Instructional Reform Strategies, by 
Subtopic 

Using Data to Identify and Implement an Instructional Program 

Using data to evaluate instructional programs (for example, measuring program effectiveness) 

English language arts or math curricula were aligned with state academic standards 

Promoting the Continuous Use of Student Data 

Using data to track or monitor the performance of SIG schools or to inform and differentiate instruction 

Using interim or benchmark tests for English language arts or math 

Providing Supports and Professional Development to Staff to Assist ELLs and Students with Disabilities 

Implementing strategies (including additional supports or professional development) to ensure that limited English 
proficient students acquire language skills to master academic content 

Providing additional supports and programs to students with disabilities 

Tailoring Strategies for Secondary Schools 

Using data to track attendance, graduation rates, or student progress toward grade promotion or graduation 
Source: SIG application; interviews with district administrators in spring 2012. 

ELLs = English language learners. 

Figure B.l displays results of the analysis on the extent to which district administrators 
reported using the comprehensive instructional refonn strategies aligned with the SIG 
application criteria. Figure B.2 displays the extent to which districts reported using the individual 
practices included in the analysis for this area. Figures B.3-B.6 display the results for each 
subtopic. 
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Figure B.l. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on 
Comprehensive Instructional Reform Strategies, Spring 2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table B.l. Each dot in this figure 

represents the districts that reported using a particular number of practices (out of seven 
examined) that were aligned with the SIG application criteria. The number inside each dot is 
the number of districts represented by the dot; dots that represent less than 10 districts have 
no number inside. For two of the practices, a “yes” response received one point. In the other 
five cases, it was possible for a district to receive a fraction of one point. See Chapter II for 
details on the way in which the number of practices was determined for each district. The 
dashed line denotes the average number of practices for each group of districts. 
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Figure B.2. Study Districts’ Reported Usage of Individual Practices Aligned with SIG Objectives on 
Comprehensive Instructional Reform Strategies, Spring 2012 
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Source: Interviews with district administrators in spring 2012. 

Note: This figure has a separate panel for each subtopic. We selected district interview questions 

that aligned with the practices described in the SIG application criteria. The practices shown 
on the horizontal axis of this figure are listed in Table B.l. For each practice in the SIG 
application criteria for which we identified one or more interview questions aligned with the 
practice, we constructed a variable ranging from zero to one, with a value of one indicating 
that the district responded “yes” to all the interview questions selected for that practice. The 
height of each bar represents the mean value of the practice (on a scale of zero to one). For 
some of the practices shown in this figure, multiple interview questions aligned with that 
practice. In the figure, we indicate this using the words “extent to which” at the beginning of 
the practice, to emphasize that the level of usage of that practice is measured using multiple 
interview questions (as opposed to a single, binary measure of whether that practice was 
used). 

ELLs = English language learners; PD = professional development. 
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Figure B.3. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on 
Comprehensive Instructional Reform Strategies, Using Data to Identify and Implement an 
Instructional Program Subtopic, Spring 2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table B.l. Each dot in this figure 

represents the districts that reported using a particular number of practices (out of two 
examined) that were aligned with the SIG application criteria. The number inside each dot is 
the number of districts represented by the dot; dots that represent less than 10 districts have 
no number inside. For one practice, a “yes” response received one point. For the other, it was 
possible for a district to receive a fraction of one point. See Chapter II for details on the way 
in which the number of practices was determined for each district. The dashed line denotes 
the average number of practices for each group of districts. 
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Figure B.4. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on 
Comprehensive Instructional Reform Strategies, Promoting the Continuous Use of Student Data 
Subtopic, Spring 2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table B.l. Each dot in this figure 

represents the districts that reported using a particular number of practices (out of two 
examined) that were aligned with the SIG application criteria. The number inside each dot is 
the number of districts represented by the dot; dots that represent less than 10 districts have 
no number inside. For both strategies, it was possible for a district to receive a fraction of one 
point. See Chapter II for details on the way in which the number of practices was determined 
for each district. The dashed line denotes the average number of practices for each group of 
districts. 
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Figure B.5. Study District’s Reported Usage of Practices Aligned with SIG Objectives on 
Comprehensive Instructional Reform Strategies, Providing Supports and Professional 
Development to Staff to Assist English Language Learners and Students With Disabilities 
Subtopic, Spring 2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table B.l. Each dot in this figure 

represents the districts that reported using a particular number of practices (out of two 
examined) that were aligned with the SIG application criteria. The number inside each dot is 
the number of districts represented by the dot; dots that represent less than 10 districts have 
no number inside. For one strategy, a “yes” response received one point. For the other, it was 
possible for a district to receive a fraction of one point. See Chapter II for details on the way 
in which the number of practices was determined for each district. The dashed line denotes 
the average number of practices for each group of districts. 

ELLs = English language learners; PD = professional development. 
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Figure B.6. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on 
Comprehensive Instructional Reform Strategies, Tailoring Strategies for Secondary Schools 
Subtopic, Spring 2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practice summarized in this figure is presented in Table B.l. This figure presents one 

practice described in the SIG application criteria to which multiple interview questions 
aligned. As described in Chapter II, whenever multiple interview questions aligned with a 
single practice from the application criteria, we used those questions to construct a variable 
ranging from zero to one, with districts receiving a fraction of a point for each question to 
which they responded “yes.” Each dot in this figure represents the districts that reported using 
a particular proportion of the interview questions aligned to the practice described in the SIG 
application criteria. The number inside each dot is the number of districts represented by the 
dot; dots that represent less than 10 districts have no number inside. The dashed line 
denotes the average value for each group of districts. 
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B. Teacher and Principal Effectiveness 

We identified 10 practices from the spring 2012 district interview aligned with SIG 
objectives on teacher and principal effectiveness (see Table B.2). 

Table B.2. Practices Aligned with SIG Objectives on Teacher and Principal Effectiveness, by Subtopic 

TEACHER EFFECTIVENESS 

Using Rigorous, Transparent, and Equitable Evaluation Systems 

Requiring schools to use student achievement growth to evaluate teachers, specifying the extent to which student 
achievement growth must factor into teacher evaluations, or using state test scores to assess student growth for 
teacher evaluations 

Requiring multiple performance measures for teacher evaluations 

Identifying and Rewarding Effective Teachers and Removing Ineffective Ones 
Using data to inform decisions such as tenure, retention, and bonuses for teachers 

Providing High-Quality, Job Embedded Professional Development or Supports 

Using data to inform professional development offerings for teachers 

Using data to evaluate the success of professional development offerings for teachers 

Implementing Strategies to Recruit, Place, and Retain Staff 

Implementing strategies, such as financial incentives or induction support for novice teachers, designed to recruit, 
place, and retain teachers in SIG schools 

Modifying teacher tenure rules that affect placement in or removal from SIG schools or permitting principal 
discretion in hiring teachers for SIG schools 

PRINCIPAL EFFECTIVENESS 

Using Rigorous, Transparent, and Equitable Evaluation Systems 

Requiring schools to use student achievement growth to evaluate principals or using state test scores to assess 
student achievement growth for principal evaluations 

Requiring multiple performance measures other than student growth for principal evaluations 

Implementing Strategies to Recruit, Place, and Retain Staff 

Implementing strategies, such as financial incentives, that are designed to recruit, place, and retain principals in 
SIG schools 

Source: SIG application; interviews with district administrators in spring 2012. 

Figure B.7 displays results of the analysis on the extent to which district administrators 
reported using the teacher and principal effectiveness practices aligned with the SIG application 
criteria. Figures B.8 and B.9 display the extent to which districts reported using the individual 
teacher and principal effectiveness practices included in the analysis. Figures B.10-B.13 display 
the results for each subtopic. 
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Figure B.7. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on Teacher 
and Principal Effectiveness, Spring 2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table B.2. Each dot in this figure 

represents the districts that reported using a particular number of practices (out of 10 
examined) that were aligned with the SIG application criteria. Each dot in this figure 
represents less than 10 districts, so the numbers inside the dots have been removed to 
protect respondent confidentiality. For three of the practices, a “yes” response received one 
point. For the other seven practices, it was possible for a district to receive a fraction of one 
point. See Chapter II for details on the way in which the number of practices was determined 
for each district. The dashed line denotes the average number of practices for each group of 
districts. 
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Figure B.8. Study Districts’ Reported Usage of Individual Practices Aligned with SIG Objectives on 
Teacher and Principal Effectiveness; Using Rigorous, Transparent, and Equitable Evaluation 
Systems Subtopic and Identifying and Rewarding or Removing Teachers and Principals Subtopic, 
Spring 2012 


8 i.o- 


o 

0,0.9- 

.Q 

o 

I 0.8 - 


w 


: 0.7- 


■o 0.6- 
<u 

.2 0.4- 

tj 

ro 

qI OS- 
'S 

0 ) 0 . 2 - 

> 0.1 - 

c 
ro 
a) 


0 . 0 - 


Using Rigorous, Transparent, and Equitable Evaluation Systems 





Identifying 
and Rewarding 
or Removing 
Teachers and 
Principals 


Extent to Which 
District Requires Student 
Achievement Growth 
in Teacher Evaluations 


Extent to Which 
District Requires 
Multiple Teacher 
Evaluation Measures 


Extent to Which 
District Requires Student 
Achievement Growth 
in Principal Evaluations 


Extent to Which 
District Requires 
Multiple Principal 
Evaluation Measures 
other than 
Student Growth 

Practice Aligned with SIG Objectives 


Data Informs 
Tenure, Retention, 
or Bonuses 


I Districts (N=60) 


Source: Interviews with district administrators in spring 2012. 

Note: This figure has a separate panel for each subtopic. We selected district interview questions 

that aligned with the practices described in the SIG application criteria. The practices shown 
on the horizontal axis of this figure are listed in Table B.2. For each practice in the SIG 
application criteria for which we identified one or more interview questions aligned with the 
practice, we constructed a variable ranging from zero to one, with a value of one indicating 
that the district responded “yes” to all the interview questions selected for that practice. The 
height of each bar represents the mean value of the practice (on a scale of zero to one). For 
some of the practices shown in this figure, multiple interview questions aligned with that 
practice. In the figure, we indicate this using the words “extent to which” at the beginning of 
the practice, to emphasize that the level of usage of that practice is measured using multiple 
interview questions (as opposed to a single, binary measure of whether that practice was 
used). 
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Figure B.9. Study Districts’ Reported Usage of Individual Practices Aligned with SIG Objectives on 
Teacher and Principal Effectiveness; Providing High-Quality Job-Embedded Professional 
Development or Supports Subtopic and Implementing Strategies to Recruit, Place, and Retain 
Staff Subtopic, Spring 2012 


Providing High-Quality, 

Job Embedded PD or Supports 


Implementing Strategies to Recruit, Place, and Retain Staff 



Data Informs PD Use Data to Evaluate PD 


Extent to Which District Extent to 

Uses Financial Which District Provides 


Incentives and Flexibility for 

Other Strategies to Hiring Decisions 
Recruit and Retain 
Effective Teachers 

Practice Aligned with SIG Objectives 


Extent to 
Which District 
Uses Financial 
Incentives to 
Recruit and Retain 
Effective Principals 


Districts (N=60) 


Source: Interviews with district administrators in spring 2012. 

Note: This figure has a separate panel for each subtopic. We selected district interview questions 

that aligned with the practices described in the SIG application criteria. The practices shown 
on the horizontal axis of this figure are listed in Table B.2. For each practice in the SIG 
application criteria for which we identified one or more interview questions aligned with the 
practice, we constructed a variable ranging from zero to one, with a value of one indicating 
that the district responded “yes” to all the interview questions selected for that practice. The 
height of each bar represents the mean value of the practice (on a scale of zero to one). For 
some of the practices shown in this figure, multiple interview questions aligned with that 
practice. In the figure, we indicate this using the words “extent to which” at the beginning of 
the practice, to emphasize that the level of usage of that practice is measured using multiple 
interview questions (as opposed to a single, binary measure of whether that practice was 
used). 

PD = professional development. 
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Figure B.10. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on Teacher 
and Principal Effectiveness, Using Rigorous, Transparent, and Equitable Evaluation Systems 
Subtopic, Spring 2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table B.2. Each dot in this figure 

represents the districts that reported using a particular number of practices (out of four 
examined) that were aligned with the SIG application criteria. Each dot in this figure 
represents less than 10 districts, so the numbers inside the dots have been removed to 
protect respondent confidentiality. For all four practices, it was possible for a district to 
receive a fraction of one point. See Chapter II for details on the way in which the number of 
practices was determined for each district. The dashed line denotes the average number of 
practices for each group of districts. 
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Figure B.11. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on Teacher 
and Principal Effectiveness, Identifying and Rewarding Effective Teachers and Principals and 
Removing Ineffective Ones Subtopic, Spring 2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practice summarized in this figure is presented in Table B.2. Each dot in this figure 

represents the districts that reported using the one practice that was aligned with the SIG 
application criteria. The number inside each dot is the number of districts represented by the 
dot. To protect respondent confidentiality, the number inside the smallest dot has been 
removed. For this practice, a “yes” response received one point. See Chapter II for details on 
the way in which the number of practices was determined for each district. The dashed line 
denotes the average number of practices for each group of districts. 
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Figure B.12. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on Teacher 
and Principal Effectiveness, Providing High Quality, Job-Embedded Professional Development or 
Supports Subtopic, Spring 2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table B.2. Each dot in this figure 

represents the districts that reported using a particular number of practices (out of two 
examined) that were aligned with the SIG application criteria. The number inside each dot is 
the number of districts represented by the dot; dots that represent less than 10 districts have 
no number inside. For both practices, a “yes” response received one point. See Chapter II for 
details on the way in which the number of practices was determined for each district. The 
dashed line denotes the average number of practices for each group of districts. 

PD = professional development. 
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Figure B.13. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on Teacher 
and Principal Effectiveness, Implementing Strategies to Recruit, Place, and Retain Staff Subtopic, 
Spring 2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table B.2. Each dot in this figure 

represents the districts that reported using a particular number of practices (out of three 
examined) that were aligned with the SIG application criteria. Each dot in this figure 
represents less than 10 districts, so the numbers inside the dots have been removed to 
protect respondent confidentiality. For all three practices, it was possible for a district to 
receive a fraction of one point. See Chapter II for details on the way in which the number of 
practices was determined for each district. The dashed line denotes the average number of 
practices for each group of districts. 
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C. Learning Time and Community-Oriented Schools 

We identified two practices from the spring 2012 district interview aligned with SIG 
objectives on learning time and community-oriented schools (see Table B.3). 

Table B.3. Practices Aligned with SIG Objectives on Learning Time and Community-Oriented Schools, by 
Subtopic 

Increasing Learning Time 

Increasing the minimum amount of time spent each week on English language arts or math instruction or 
increasing the number of instructional days in the school year 

Engaging Families and Communities 

Using data to guide the development and implementation of nonacademic supports or enrichment programs, for 
example, to identify how many and which students need counseling 

Source: SIG application; interviews with district administrators in spring 2012. 

Figure B.14 displays results of the analysis on the extent to which district administrators 
reported using the increasing learning time and creating community-oriented schools practices 
aligned with the SIG application criteria. Figure B.15 displays the extent to which districts 
reported using the individual practices included in the analysis for this area. Figures B.16 and 
B.17 display the results for each subtopic. 
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Figure B.14. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on Learning 
Time and Community-Oriented Schools, Spring 2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table B.3. Each dot in this figure 

represents the districts that reported using a particular number of practices (out of two 
examined) that were aligned with the SIG application criteria. The number inside each dot is 
the number of districts represented by the dot; dots that represent less than 10 districts have 
no number inside. For one practice, a “yes” response received one point. For the other 
practice, it was possible for a district to receive a fraction of one point. See Chapter II for 
details on the way in which the number of practices was determined for each district. The 
dashed line denotes the average number of practices for each group of districts. 
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Figure B.15. Study Districts’ Reported Usage of Individual Practices Aligned with SIG Objectives 
on Learning Time and Community-Oriented Schools, Spring 2012 



Extent to Which Use Data to Guide 

District Increases Time Development of 

Spent Teaching and Nonacademic Supports 

Number of or Enrichment Programs 

Instructional Days 

Practice Aligned with SIG Objectives 

| Districts (N=60) 


Source: Interviews with district administrators in spring 2012. 

Note: This figure has a separate panel for each subtopic. We selected district interview questions 

that aligned with the practices described in the SIG application criteria. The practices shown 
on the horizontal axis of this figure are listed in Table B.3. For each practice in the SIG 
application criteria for which we identified one or more interview questions aligned with the 
practice, we constructed a variable ranging from zero to one, with a value of one indicating 
that the district responded “yes” to all the interview questions selected for that practice. The 
height of each bar represents the mean value of the practice (on a scale of zero to one). For 
some of the practices shown in this figure, multiple interview questions aligned with that 
practice. In the figure, we indicate this using the words “extent to which” at the beginning of 
the practice, to emphasize that the level of usage of that practice is measured using multiple 
interview questions (as opposed to a single, binary measure of whether that practice was 
used). 
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Figure B.16. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on Learning 
Time and Community-Oriented Schools, Increasing Learning Time Subtopic, Spring 2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practice summarized in this figure is presented in Table B.3. This figure presents one 

practice described in the SIG application criteria to which multiple interview questions 
aligned. As described in Chapter II, whenever multiple interview questions aligned with a 
single practice from the application criteria, we used those questions to construct a variable 
ranging from zero to one, with districts receiving a fraction of a point for each question to 
which they responded “yes.” Each dot in this figure represents the districts that reported using 
a particular proportion of the interview questions aligned to the practice described in the SIG 
application criteria. The number inside each dot is the number of districts represented by the 
dot; dots that represent less than 10 districts have no number inside. The dashed line 
denotes the average value for each group of districts. 
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Figure B.17. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on Learning 
Time and Community-Oriented Schools, Engaging Families and Communities Subtopic, Spring 
2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practice summarized in this figure is presented in Table B.3. Each dot in this figure 

represents the districts that reported using the one practice that was aligned with the SIG 
application criteria. The number inside each dot is the number of districts represented by the 
dot. To protect respondent confidentiality, the number inside the smallest dot has been 
removed. For this practice, a “yes” response received one point. See Chapter II for details on 
the way in which the number of practices was determined for each district. The dashed line 
denotes the average number of practices for each group of districts. 
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D. Operational Flexibility and Support 

We identified three practices from the spring 2012 district interview aligned with SIG 
objectives on operational flexibility and support (see Table B.4). 

Table B.4. Practices Aligned with SIG Objectives on Operational Flexibility and Support, by Subtopic 

Providing Operational Flexibility 

Low-performing schools had primary responsibility for budget, hiring, discipline, or school year length decisions 

Receiving Technical Assistance and Support 

Receiving training, technical assistance, or access to data from the state to support school improvement efforts or 
use data to improve instruction 

Having a designated office or staff or contracting with external consultants to support school turnaround efforts 
Source: SIG application; interviews with district administrators in spring 2012. 

Figure B.18 displays results of the analysis on the extent to which district administrators 
reported using the operational flexibility and support practices aligned with the SIG application 
criteria. Figure B.19 displays the extent to which districts reported using the individual 
operational flexibility and support practices included in the analysis. Figures B.20 and B.21 
display the results for each subtopic. 
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Figure B.18. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on 
Operational Flexibility and Support, Spring 2012 
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Districts (N=60) 


Source: Interviews with district administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table B.4. Each dot in this figure 

represents the districts that reported using a particular number of practices (out of three 
examined) that were aligned with the SIG application criteria. Each dot in this figure 
represents less than 10 districts, so the numbers inside the dots have been removed to 
protect respondent confidentiality. For all three practices, it was possible for a district to 
receive a fraction of one point. See Chapter II for details on the way in which the number of 
practices was determined for each district. The dashed line denotes the average number of 
practices for each group of districts. 
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Figure B.19. Study Districts’ Reported Usage of Individual Practices Aligned with SIG Objectives 
on Operational Flexibility and Support, Spring 2012 



Extent to Which 
District Provides Autonomy 
to Schools on 
Budgeting, Hiring, 
Discipline, or 
School Year Length 


Extent to Which 
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to Support School Improvement 
or Use Data to 
Improve Instruction 

Practice Aligned with SIG Objectives 


Extent to Which 
District Provides 
Governance Structure 
to Support School 
Turnaround 


I Districts (N=60) 


Source: Interviews with district administrators in spring 2012. 

Note: This figure has a separate panel for each subtopic. We selected district interview questions 

that aligned with the practices described in the SIG application criteria. The practices shown 
on the horizontal axis of this figure are listed in Table B.4. For each practice in the SIG 
application criteria for which we identified one or more interview questions aligned with the 
practice, we constructed a variable ranging from zero to one, with a value of one indicating 
that the district responded “yes” to all the interview questions selected for that practice. The 
height of each bar represents the mean value of the practice (on a scale of zero to one). For 
some of the practices shown in this figure, multiple interview questions aligned with that 
practice. In the figure, we indicate this using the words “extent to which” at the beginning of 
the practice, to emphasize that the level of usage of that practice is measured using multiple 
interview questions (as opposed to a single, binary measure of whether that practice was 
used). 
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Figure B.20. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on 
Operational Flexibility and Support, Providing Operational Flexibility Subtopic, Spring 2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practice summarized in this figure is presented in Table B.4. This figure presents one 

practice described in the SIG application criteria to which multiple interview questions 
aligned. As described in Chapter II, whenever multiple interview questions aligned with a 
single practice from the application criteria, we used those questions to construct a variable 
ranging from zero to one, with districts receiving a fraction of a point for each question to 
which they responded “yes.” Each dot in this figure represents the districts that reported using 
a particular proportion of the interview questions aligned to the practice described in the SIG 
application criteria. The number inside each dot is the number of districts represented by the 
dot; dots that represent less than 10 districts have no number inside. The dashed line 
denotes the average value for each group of districts. 
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Figure B.21. Study Districts’ Reported Usage of Practices Aligned with SIG Objectives on 
Operational Flexibility and Support, Receiving Technical Assistance and Support Subtopic, Spring 
2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table B.4. Each dot in this figure 

represents the districts that reported using a particular number of practices (out of two 
examined) that were aligned with the SIG application criteria. Each dot in this figure 
represents less than 10 districts, so the numbers inside the dots have been removed to 
protect respondent confidentiality. For both practices, it was possible for a district to receive a 
fraction of one point. See Chapter II for details on the way in which the number of practices 
was determined for each district. The dashed line denotes the average number of practices 
for each group of districts. 
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In contrast to the main body of the report (which summarized the extent to which schools 
reported using the practices promoted by School Improvement Grants [SIG] and presented 
figures showing the extent to which schools reported using the individual practices within each 
topic area and subtopic), and Appendix B (which summarized the extent to which districts 
reported using the practices promoted by SIG and presented figures showing the extent to which 
districts reported using the individual practices within each topic area and subtopic), this 
appendix presents detailed findings from individual interview and survey questions, and 
describes how we analyzed those data. Specifically, we show the number of states and the 
percentage of districts, schools implementing a SIG-funded intervention model, and schools not 
implementing a SIG-funded model that responded “yes” to each question examined as part of 
this report. Readers interested in responses to individual interview and survey questions may, 
therefore, find this appendix useful. 

The school-level data presented in this appendix are the same data used for the analyses 
presented in Chapter IV. The school-level tables in this appendix present results separately for 
schools implementing a SIG-funded model and schools not implementing a SIG-funded model. 

The state- and district-level data presented in this appendix came from structured telephone 
interviews with administrators in the 60 districts and 22 states where the SIG-sample schools 
were located. These interviews, conducted in spring 2012, documented the state- and district- 
level supports for the school turnaround practices used by schools. The overarching research 
question answered by these findings is: How are states and districts supporting schools’ efforts to 
use practices promoted by SIG? All 60 districts and 22 states in the SIG study sample included 
schools that were and were not implementing a SIG-funded model. Therefore, the state- and 
district-level tables in this appendix do not present comparisons; instead, they present descriptive 
information about the practices that districts and states reported using. 

In Section A, we discuss how we analyzed data from closed- and open-ended questions and 
how we handled missing values. In Section B, we present findings from the interview questions 
in a series of tables, the titles of which are shown in the list of tables at the beginning of this 
report. 

A. Analysis Methods 

Analyzing data from closed-ended questions. The evaluation’s interviews and surveys 
comprised mostly closed-ended questions — that is, questions with yes-or-no responses or with a 
set of specific response categories from which to choose. As a result, these variables were 
already in a fonnat that is suitable, or nearly suitable, for analysis. 

Closed-ended questions sometimes included an “other-specify” response option so the 
interview or survey could progress smoothly when a respondent was uncertain about the 
response option that applied or could not find a response option that adequately captured the 
response he or she wished to provide. When a respondent chose this option, the interviewer 
asked the respondent to specify his or her response and recorded it. These “other-specify” 
responses were reviewed and either recoded into one of the existing structured response 
categories or coded into new response categories, as appropriate. Following reporting 
requirements established by the U.S. Department of Education’s National Center for Education 
Statistics, we created a new response category only if at least three respondents (that is, states, 
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districts, or schools) provided the same or similar response. If fewer than three respondents 
provided a particular response, the response remained part of the broad “other” category. 

Analyzing data from open-ended questions. Whenever possible, we categorized the 
responses to open-ended questions into nominal categories (based on the themes that emerged) 
that could then be treated as quantitative, categorical data. This strategy enabled us to 
systematically identify and report on recurring themes mentioned frequently by respondents. 

Handling missing values. Values can be missing for various reasons: (1) because the 
respondent did not complete the interview or survey; (2) because the respondent completed the 
interview or survey but did not complete the question; (3) because the respondent chose “don’t 
know,” “refused,” or “not applicable”; or (4) because the question was logically skipped based 
on earlier responses. Generally, we excluded all missing values from our calculations regardless 
of the reason that the question was missing (that is, we did not recode a missing as a zero). 1 In 
the tables presented in this appendix, we report the sample sizes for states, districts, and schools 
with nonmissing values on the given item. Percentages generally total 100 percent. 2 

Selecting survey questions aligned with the SIG application criteria. We reviewed the 
school survey questions and assigned those that aligned with the practices described in the SIG 
application criteria to specific topic areas and subtopics. We determined the subtopic into which 
each survey question fell based on the section of the SIG application criteria with which it 
aligned. In the tables presented in Section B, the last column of each table indicates whether each 
question was selected, and if it was selected, for which subtopic, by using the abbreviations 
shown in Table C. 1 . 


Table C.1 . Abbreviations for Subtopics 


Subtopic 

Abbreviation 

Topic Area: Comprehensive Instructional Reform Strategies 

Using data to identify and implement an instructional program 

IS-1 

Promoting the continuous use of student data 

IS-2 

Providing supports and professional development to staff to assist both English 
language learners and students with disabilities 

IS-3 

Using and integrating technology-based supports 

ISA 

Tailoring strategies for secondary schools 

IS-5 

Topic Area: Teacher and Principal Effectiveness 

Using rigorous, transparent, and equitable evaluation systems 

TL-1 

Identifying and rewarding effective teachers and principals and removing ineffective 
ones 

TL-2 

Providing high quality, job-embedded professional development or supports 

TL-3 

Implementing strategies to recruit, place, and retain staff 

TL-4 


1 The occasional instances in which we recoded missing values as zeros are footnoted in the tables. 

2 In some cases, the number of states in a table totals more than 22 or the percentage of districts or schools 
totals more than 100 percent; we include a footnote to those tables explaining why. As one example, if the question 
asked the respondent to mark all responses that apply, respondents could choose multiple answers. 
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Subtopic 

Abbreviation 

Topic Area: Learning Time and Community-Oriented Schools 

Increasing learning time 

TC-1 

Engaging families and communities and providing a safe school environment that 


meets students’ social, emotional, and health needs 

TC-2 

Topic Area: Operational Flexibility and Support 

Having operational flexibility 

FS-1 

Receiving technical assistance and support 

FS-2 


Source: Surveys of school administrators in spring 2012. 


B. Detailed Findings from Interview and Survey Questions 

In this section we present findings from particular interview and survey questions based on 
U.S. Department of Education guidance about those in which it had the most interest. The tables 
are organized to follow the order of the modules in the interview and survey protocols, which 
was: (1) data systems, (2) teachers and leaders, (3) school turnaround, and (4) charter schools (no 
questions from the standards and assessments module are presented here). 


Table C.2. District Reports of Their Schools’ Access to Statewide Longitudinal Data Systems and District 
Data Systems, Spring 2012 


Item Aligned with 
SIG Application 
Percentage of Criteria 

Districts (Subtopic) 


Reported that schools in their district have: 

Both direct access to the SLDS data and access to district- or state- 
generated reports based on SLDS data 64.3 

Only direct access to SLDS data 0.0 

Only access to district- or state-generated reports based on SLDS data 25.0 

Access to neither type of information 1 0.7 

Reported that schools in their district have access to data from a district data 
system 3 that is distinct from the SLDS: 

Both direct access to the district data and access to district-generated 
reports based on district data 91 .5 

Only direct access to the district data, only access to district-generated 
reports based on district data, or access to neither type of information 845 

Number of Districts 60 


Yes (FS-2) 


Yes (FS-2) 


Source: Interviews with district administrators in spring 2012. 

a District data system(s) (also identified as local instructional improvement systems) are defined by the U.S. 
Department of Education as technology-based tools and other strategies that provide teachers, principals, and 
administrators with meaningful support and actionable data to systemically manage continuous instructional 
improvement. 

b To comply with NCES statistical reporting requirements for small cell sizes, we aggregated the percentages for 
“only direct access to the district data,” “only access to district-generated reports based on district data,” and “access 
to neither type of information.” 

SLDS = Statewide Longitudinal Data Systems; FS-2 = Receiving technical assistance and support. 
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Table C.3. District Use of Data Analysis to Monitor SIG School Performance, Spring 2012 



Percentage of Districts 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported tracking or monitoring the performance 
of SIG grantees implementing one of the four 
SIG intervention models by: 

Analyzing student achievement by grade level 
and/or by subject, by school 

100.0 

Yes (IS-2) 

Analyzing student achievement data over time 
to identify trends 

100.0 

Yes (IS-2) 

Examining other measures of student 
progress, such as benchmarks or diagnostic 
tests 

96.6 

Yes (IS-2) 

Examining achievement gaps between groups 
of students, such as NCLB subgroups 

96.6 

Yes (IS-2) 

Tracking graduation rates 

96.3 

Yes (IS-5) 

Tracking student readiness for grade 
promotion or graduation 

91.4 

Yes (IS-5) 

Tracking students’ postsecondary enrollment 
and progress 

54.7 

Yes (IS-5) 

Monitoring student attendance 

100.0 

Yes (IS-5) 

Other analyses 

75.9 

No 

Reported using different analyses for SIG 
schools compared with other schools in the 
district 

18.6 

No 

Number of Districts 

50-60 



Source: Interviews with district administrators in spring 2012. 

Note: A range is provided for the sample size because nonresponse varied across items. 

IS-2 = Promoting the continuous use of student data; IS-5 = Tailoring strategies for secondary schools; 
NCLB = No Child Left Behind. 
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Table C.4. Purposes for Which District Staff Use Data, Spring 2012 



Percentage of Districts 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported using data on all students from the SLDS, a 
district data system, or both, for the following purposes: 



To track overall school performance and identify 
areas for improvement 

100.0 

Yes (IS-2) 

To evaluate instructional programs 

93.2 

Yes (IS-1) 

To guide development and implementation of 
academic supports or enrichment programs 

98.3 

Yes (IS-2) 

To guide development and implementation of 
nonacademic supports or enrichment programs 
(for example, counseling) 

81.0 

Yes (TC-2) 

To track students’ progress toward graduation 

93.1 

Yes (IS-5) 

To track students’ postsecondary enrollment and 
progress 

55.2 

No 

To inform professional development offerings for 
teachers, principals, or other school leaders 

89.8 

Yes (TL-3) 

To evaluate the success of professional development 
offerings for teachers, principals, or other school 
leaders 

64.4 

Yes (TL-3) 

To inform other decisions regarding individual 
teachers, principals, or other school leaders (such 
as tenure, retention, or bonus decisions) 

66.1 

Yes (TL-2) 

To inform resource allocation to improve instruction 

93.2 

Yes (IS-2) 

For other purposes 

39.7 

No 

Reported using data on ELLs from the SLDS, a district 
data system, or both, for the following purposes: 



To make decisions about students’ entry into and/or 
exit from ELL status 

98.3 

Yes (IS-3) 

To place ELLs into specialized programs and classes 

98.3 

Yes (IS-3) 

To track the progress of current ELLs 

98.3 

Yes (IS-3) 

To track the progress of former ELLs 

89.1 

Yes (IS-3) 

To inform, improve, or differentiate instruction for 
ELLs 

93.1 

Yes (IS-2) 

To identify professional development needs for 
teachers of ELLs 

86.0 

Yes (IS-3) 

To assess teacher effectiveness with ELLs 

57.9 

Yes (IS-3) 

For other purposes 

48.2 

No 

Number of Districts 

60 



Source: Interviews with district administrators in spring 2012. 

SLDS = Statewide Longitudinal Data System; ELLs = English language learners; IS-1 = Using data to identify and 
implement an instructional program; IS-2 = Promoting the continuous use of student data; IS-3 = Providing supports 
and professional development to staff to assist ELLs and students with disabilities; IS-5 = Tailoring strategies for 
secondary schools; TC-2 = Engaging families and communities and providing a safe school environment that meets 
students’ social, emotional, and health needs; TL-2 = Identifying and rewarding effective teachers and principals and 
removing ineffective ones; TL-3 = Providing high-quality, job-embedded professional development or supports. 
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Table C.5. Purposes for Which School Staff Use Data, Spring 2012 



Percentage of Low-Performing Schools 



Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Not Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported using data for the following purposes: 




To evaluate instructional programs 

96.2 

92.0 

Yes (IS-1) 

To guide development and implementation of 
academic supports or enrichment programs 

99.3 

98.3 

Yes (IS-2) 

To guide development and implementation of 
nonacademic supports or enrichment programs 
(for example, counseling) 

89.6 

83.9 

Yes (TC-2) 

To inform teachers’ instructional practices 

98.6 

96.5 

Yes (IS-2) 

To inform professional development offerings 

96.5 

91.9 

Yes (TL-3) 

To evaluate the success of professional 
development offerings 

80.5 

73.8 

Yes (TL-3) 

To track individual student performance and 
identify areas of improvement for specific 
students 

98.6 

99.4 

Yes (IS-2) 

To track students’ progress toward high school 
graduation 3 

97.8 

98.8 

Yes (IS-5) 

To track students’ preparation for college 
enrollment 3 

89.8 

93.8 

Yes (IS-5) 

To track students’ postsecondary enrollment and 
progress 3 

72.3 

85.4 

No 

To inform resource allocation to improve 
instruction 

88.8 

82.6 

Yes (IS-2) 

Other purpose 

12.4 

13.1 

No 

Among schools that reported having ELLs, reported 
using data on ELL for the following purposes: 




To make decisions about students’ entry into 
and/or exit from ELL status 

95.4 

94.0 

Yes (IS-3) 

To place ELLs into specialized programs and 
classes 

93.2 

89.5 

Yes (IS-3) 

To track the progress of current ELLs 

98.0 

94.0 

Yes (IS-3) 

To track the progress of former ELLs 

76.6 

64.6 

Yes (IS-3) 

To inform, improve, or differentiate instruction for 
ELLs 

92.8 

89.4 

Yes (IS-2) 

To identify professional development needs for 
teachers of ELLs 

81.0 

73.5 

Yes (IS-3) 

To assess teacher effectiveness with ELLs 

79.5 

73.1 

Yes (IS-3) 

Other purpose 

7.5 

_b 

No 

Number of Schools 

140-290 

80-170 



Source: Surveys of school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a The analysis for this row includes only high schools. 
b This cell has been suppressed to protect respondent confidentiality. 

ELLs = English language learners; IS-1 = Using data to identify and implement an instructional program; IS-2 = 
Promoting the continuous use of student data; IS-3 = Providing supports and professional development to staff to 
assist ELLs and students with disabilities; IS-5 = Tailoring strategies for secondary schools; TC-2 = Engaging families 
and communities and providing a safe school environment that meets students’ social, emotional, and health needs; 
TL-3 = Providing high-quality, job-embedded professional development or supports. 
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Table C.6. Supports for Data Use, Spring 2012 


- 

Percentage of Low-Performing Schools 

Implementing Not Implementing 

a SIG-Funded a SIG-Funded 

Intervention Model Intervention Model 

in 2011-2012 in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported receiving the following support 
to help school staff access and use data: 
Funds to support school investments related 
to data use 

69.3 

38.7 

Yes (FS-2) 

Hardware or software to facilitate data use 

56.5 

36.3 

Yes (FS-2) 

Materials on how to access and use data to 
differentiate or improve instruction 

56.9 

42.4 

Yes (FS-2) 

Other type of support 

15.0 

10.5 

No 

Reported having a designated staff person 
who supports the use of data by teachers 

92.3 

85.1 

No 

Reported providing scheduled time for 
teachers to examine data, either on their own 
or in collaboration with others 

96.8 

94.8 

Yes (TL-3) 

Reported that their school leaders coached 
teachers on the use of data to: 

Improve instruction 

98.3 

95.9 

Yes (TL-3) 

Improve instruction of ELLs 

77.2 

72.4 

Yes (IS-3) 

Reported receiving professional development, 
training, or technical assistance to help school 
staff access data, navigate data systems, or 
interpret and use data 

90.2 

85.5 

Yes (TL-3) 

Average reported number of hours this 
professional development, training, or technical 
assistance was provided to: a 
School administrators 

19.0 

14.4 

No 

Teachers 

25.1 

15.3 

No 

Among schools that reported having ELLs, 
reported receiving the following supports to 
help school staff access and use data related 
to ELLs: 

Supports to use data to track the 
performance of ELLs 

60.8 

55.1 

Yes (IS-3) 

Supports to use data to improve or 
differentiate instruction for ELLs 

59.4 

55.1 

Yes (IS-3) 

Other supports to use data about ELLs 

35.6 

29.4 

No 

Number of Schools 

190-290 

130-170 



Source: Surveys of school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Schools that reported they did not receive professional development, training, or technical assistance to help school 
administrators and/or teachers access data, navigate data systems, or interpret and use data to improve and/or 
differentiate instruction are included in the analysis of this question as “no” responses. 

ELLs = English language learners; FS-2 = Receiving technical assistance and support; IS-3 = Providing supports and 
professional development to staff to assist ELLs and students with disabilities; TL-3 = Providing high-quality, job- 
embedded professional development or supports. 
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Table C.7. District Requirements for Teacher Evaluations, Spring 2012 



Percentage of 
Districts 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported that all schools must use the same teacher 

evaluation model 

89.8 

No 

Reported that student achievement growth was required 

47.5 

Yes (TL-1) 

Reported that student achievement growth was required 

with the following weight 3 


Yes (TL-1) 

No specific weight required or did not require student 

achievement growth, or other b 

81.0 


1-20 

6.9 


21-34 

0.0 


35-50 

12.1 


51 or more 

0.0 


“Significant”, “Substantial,” or “Primary” factor 

0.0 


Reported using the following number of rating levels for 

overall teacher evaluations 


No 

Four or more 

57.6 


Three rating levels 

20.3 


Two rating levels 

22.0 


Number of Districts 

60 



Source: Interviews with district administrators in spring 2012. 

a Districts that responded no to a question about whether the district required student achievement growth 
are included in the analysis of this question as “no” responses. 

b To comply with NCES statistical reporting requirements for small cell sizes, we aggregated the 
percentages for “no specific weight required or did not require student achievement growth” and “other.” 

TL-1 = Using rigorous, transparent, and equitable evaluation systems. 
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Table C.8. District-Reported Requirements for Performance Measures (Other than Student 
Achievement Growth) for Evaluations of Teachers in Tested Grades and/or Subjects 



Percentage of Districts 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Classroom observations 


Yes (TL-1) 

2009-2010 

93.2 


2011-2012 

93.2 


Self-assessment 


Yes (TL-1) 

2009-2010 

32.2 


2011-2012 

42.4 


Portfolios or other artifacts of teacher practice 


Yes (TL-1) 

2009-2010 

27.1 


2011-2012 

32.2 


Peer assessments other than classroom 
observations 


Yes (TL-1) 

2009-2010 

_a 


2011-2012 

_a 


Student work samples 


Yes (TL-1) 

2009-2010 

16.9 


2011-2012 

25.4 


Student surveys or other feedback 


Yes (TL-1) 

2009-2010 

_a 


2011-2012 

8.5 


Parent surveys or other feedback 


Yes (TL-1) 

2009-2010 

_a 


2011-2012 

_a 


Other measures 


No 

2009-2010 

16.9 


2011-2012 

25.4 


Number of Districts 

60 



Source: Interviews with district administrators in spring 2012. 

a This cell has been suppressed to protect respondent confidentiality. 


TL-1 = Using rigorous, transparent, and equitable evaluation systems. 
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Table C.9. District-Reported Requirements for Performance Measures (Other than Student 
Achievement Growth) for Evaluations of Teachers in Nontested Grades and Subjects 



Percentage of Districts 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Classroom observations 


Yes (TL-1 ) 

2009-2010 

93.2 


2011-2012 

93.2 


Self-assessment 


Yes (TL-1) 

2009-2010 

32.2 


2011-2012 

40.7 


Portfolios or other artifacts of teacher practice 


Yes (TL-1 ) 

2009-2010 

28.8 


2011-2012 

33.9 


Peer assessments other than classroom 
observations 


Yes (TL-1) 

2009-2010 

_a 


2011-2012 

_a 


Student work samples 


Yes (TL-1) 

2009-2010 

18.6 


2011-2012 

25.4 


Student surveys or other feedback 


Yes (TL-1) 

2009-2010 

5.1 


2011-2012 

8.5 


Parent surveys or other feedback 


Yes (TL-1) 

2009-2010 

_a 


2011-2012 

_a 


Other measures 


No 

2009-2010 

16.9 


2011-2012 

25.4 


Number of Districts 

60 



Source: Interviews with district administrators in spring 2012. 


a This cell has been suppressed to protect respondent confidentiality. 
TL-1 = Using rigorous, transparent, and equitable evaluation systems. 
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Table C.10. District-Reported Policies for Tenure and Frequency of Teacher Evaluation, Spring 
2012 


District-Reported Regulation 

Percentage of Districts 
(unless otherwise specified) 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Allow teachers to earn tenure 3 

81.4 

No 

Have a probationary period for all or 

some teachers 

96.6 

No 

Among districts with probationary 
period, reported mean duration of 

probationary period (years) 

2.8 

No 

Evaluate probationary teachers 


No 

Three or more times per year or 

other interval 13 

19.3 


Two times per year 

40.4 


Annually 

40.4 


Every other year 

0.0 


Evaluate non-probationary teachers 


No 

Three or more times per year 

5.1 


Two times per year 

10.2 


Annually 

44.1 


Every other year 

20.3 


Other interval 

20.3 


Number of Districts 

50-60 



Source: Interviews with district administrators in spring 2012. 

Note: A range is provided for the sample size because nonresponse varied across items. 

a This includes districts that provide teachers with some other continuing right to their job that the district 
does not refer to as “tenure.” 

b To comply with NCES statistical reporting requirements for small cell sizes, we aggregated the 
percentages for “three or more times per year” and “other interval.” 
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Table C.11. School-Reported Policies for Using Student Achievement Growth in Teacher 
Evaluations, Spring 2012 



Percentage of Low-Performing Schools 

Implementing Not Implementing 

a SIG-Funded a SIG-Funded 

Intervention Model Intervention Model 

in 2011-2012 in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported that student achievement 
growth was required 

53.5 

45.1 

Yes (TL-1) 

Reported that student achievement 
growth was required with a specific 
weight: 3 

No specific weight required or did 
not require student achievement 
growth 

54.6 

64.8 

Yes (TL-1) 

1-20 

7.1 

3.1 


21-34 

5.9 

3.1 


35-50 

11.5 

10.5 


51 or more, or “Significant,” 
“Substantial,” or “Primary” factor b 

5.2 

5.6 


Other 

15.6 

13.0 


Number of Schools 

270-280 

160-170 



Source: Surveys with school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Schools that responded no to a question about whether student achievement growth was required as a 
component of teacher evaluations are in included in the analysis of this question as “no” responses. 
b To comply with NCES statistical reporting requirements for small cell sizes, we aggregated the 
percentages for “51 or more” and “significant, substantial, or primary factor.” 

TL-1 = Using rigorous, transparent, and equitable evaluation systems. 
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Table C.12. School-Reported Performance Measures (Other than Student Achievement Growth) for 
Teacher Evaluations, Spring 2012 



Percentage of Low-Performing Schools 



Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Not Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Classroom observations 

98.2 

98.3 

Yes (TL-1) 

Self-assessment 

62.1 

47.7 

Yes (TL-1) 

Peer assessment 

21.5 

15.8 

Yes (TL-1) 

Portfolios or other artifacts of teacher 
practice 

44.7 

34.5 

Yes (TL-1) 

Student work samples 

47.8 

44.2 

Yes (TL-1) 

Student surveys or other feedback 

27.7 

26.9 

Yes (TL-1) 

Parent surveys or other feedback 

25.5 

24.0 

Yes (TL-1) 

Other measures 

25.6 

13.4 

No 

Number of Schools 

130-280 

80-170 



Source: Surveys with school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

TL-1 = Using rigorous, transparent, and equitable evaluation systems. 
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Table C.13. School-Reported Policies for Tenure and Frequency of Teacher Evaluation, Spring 
2012 



Percentage of Low-Performing Schools 
(unless otherwise specified) 

Implementing Not Implementing 

a SIG-Funded a SIG-Funded 

Intervention Model Intervention Model 
in 2011-2012 in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Allow teachers to earn tenure 3 

70.9 

72.6 

No 

Have a probationary period for 

teachers 

93.7 

98.6 

No 

Among schools reporting any duration 
for probationary period, mean duration 

of probationary period (years) 

2.6 

2.6 

No 

Evaluate probationary teachers 
Three or more times per year 

20.4 

16.8 

No 

Two times per year 

46.3 

43.1 


Annually 

20.7 

29.9 


Every other year 

0.0 

0.0 


Other 

12.6 

10.2 


Evaluate non probationary teachers 
Three or more times per year 

8.1 

10.2 

No 

Two times per year 

27.6 

28.7 


Annually 

35.3 

32.9 


Every other year 

17.6 

22.2 


Other 

11.4 

6.0 


Number of Schools 

220-280 

140-170 



Source: Surveys with school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a This includes schools that provide teachers with some other continuing right to their job that is not 
referred to as “tenure.” 
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Table C.14. School-Reported Uses of Teacher Evaluation Results, Spring 2012 



Percentage of Low-Performing Schools 

Implementing Not Implementing 

a SIG-Funded a SIG-Funded 

Intervention Model Intervention Model 
in 2011-2012 in 201 1-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported using teacher evaluation 
results to guide decisions about: 
Professional development and/or 
support 

85.3 

77.3 

Yes (TL-2) 

Annual salary increases 

11.3 

15.2 

Yes (TL-2) 

Bonuses or other performance- 
based compensation (other than 
annual salary increases) 

16.6 

19.3 

Yes (TL-2) 

Career-advancement opportunities 

42.3 

41.8 

Yes (TL-4) 

Reductions in force and excessing 
decisions 

25.5 

32.0 

Yes (TL-4) 

Number of Schools 

280 

170 



Source: Surveys with school administrators in spring 2012. 

TL-2 = Identifying and rewarding effective teachers and principals and removing ineffective ones; TL-4 = 
Implementing strategies to recruit, place, and retain staff. 
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Table C.15. District Principal Evaluation Requirements, Spring 2012 



Percentage of Districts 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported that all schools must use the same 
principal evaluation model 

96.6 

No 

Reported that student achievement growth was 
required 

59.3 

Yes (TL-1 ) 

Reported using the following number of rating 
categories for overall performance: 

Four or more rating levels 
Three rating levels 

Two rating levels, no rating levels, or do not 
specify minimum number of rating levels® 

67.8 
15.3 

16.9 

No 

Number of Districts 

60 



Source: Interviews with district administrators in spring 2012. 

a To comply with NCES statistical reporting requirements for small cell sizes, we aggregated the 
percentages for “2 rating levels” and “do not specify minimum number of rating levels or no rating levels.” 

TL-1 = Using rigorous, transparent, and equitable evaluation systems. 
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Table C.16. District-Reported Requirements for Performance Measures for Principal Evaluations, 
Spring 2012 



Percentage of Districts 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Student achievement growth 


Yes (TL-1) 

2009-2010 

44.1 


2011-2012 

59.3 


Self-assessment 


Yes (TL-1) 

2009-2010 

49.2 


2011-2012 

61.0 


District administrator input 


Yes (TL-1) 

2009-2010 

89.8 


2011-2012 

93.2 


Staff input 


Yes (TL-1) 

2009-2010 

15.3 


2011-2012 

20.3 


Student input 


Yes (TL-1) 

2009-2010 

6.8 


2011-2012 

13.6 


Other measures 


No 

2009-2010 

22.0 


2011-2012 

25.4 


Number of Districts 

60 



Source: Interviews with district administrators in spring 2012. 

TL-1 = Using rigorous, transparent, and equitable evaluation systems. 
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Table C.17. School-Reported Performance Measures for Principal Evaluations, Spring 2012 



Percentage of Low-Performing Schools 

Implementing Not Implementing 

a SIG-Funded a SIG-Funded 

Intervention Model Intervention Model 
in 2011-2012 in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported that student achievement 
growth was used 

75.8 

75.9 

Yes (TL-1) 

Reported that student achievement 
growth was required with the following 
weight: 

No specific weight required or did 
not require student achievement 
growth 

31.3 

32.7 

Yes (TL-1) 

1-20 

11.5 

8.2 


21-34 

3.7 

2.0 


35-50 

11.9 

10.9 


51 or more 

3.3 

7.5 


“Significant,” “Substantial,” or 
“Primary” factor 

13.2 

21.1 


Other 

25.1 

17.7 


Reporting using other measures: 
Self-assessment 

64.1 

58.6 

Yes (TL-1) 

District administrator input 

92.4 

91.1 

Yes (TL-1) 

School staff surveys or other 
feedback 

40.5 

41.6 

Yes (TL-1) 

Student surveys or other feedback 

29.2 

30.5 

Yes (TL-1) 

Other measures 

35.1 

22.2 

No 

Number of Schools 

90-280 

70-170 



Source: Surveys with school administrators in spring 2012. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

TL-1 = Using rigorous, transparent, and equitable evaluation systems. 
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Table C.18. School-Reported Uses of Principal Evaluation Results, Spring 2012 



Percentage of Low-Performing Schools 



Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Not Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported using results to guide 
decisions about: 




Professional development and/or 
support 

51.8 

45.0 

No 

Annual salary increases 

13.3 

20.1 

Yes (TL-3) 

Bonuses or other performance-based 
compensation (other than regular 
salary increases) 

16.2 

12.0 

Yes (TL-2) 

Number of Schools 

280 

170 



Source: Surveys with school administrators in spring 2012. 

TL-2 = Identifying and rewarding effective teachers and principals and removing ineffective ones; TL-3 = 
Providing high-quality, job-embedded professional development or supports. 
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Table C.19. District Use of Financial Incentives to Recruit or Retain Effective Staff in SIG Schools 
Implementing One of the SIG-Funded Intervention Models, Spring 2012 



Percentage of Districts 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported offering financial incentives 

64.4 

Yes (TL-4) 

Reported offering the following types of 
financial incentives: 3 
Signing/recruitment bonuses for: 
Teachers 

23.7 

Yes (TL-4) 

Principals 

23.7 

Yes (TL-4) 

Retention bonuses for: 
Teachers 

15.3 

Yes (TL-4) 

Principals 

11.9 

Yes (TL-4) 

Performance bonuses for: 
Teachers 

39.7 

Yes (TL-4) 

Principals 

36.2 

Yes (TL-4) 

Increased annual compensation other 
than bonuses for: 

Teachers 

28.8 

Yes (TL-4) 

Principals 

15.3 

Yes (TL-4) 

Loan forgiveness for: 
Teachers 

13.6 

Yes (TL-4) 

Principals 

6.8 

Yes (TL-4) 

Tuition reimbursement for: 
Teachers 

20.3 

Yes (TL-4) 

Principals 

11.9 

Yes (TL-4) 

Housing (purchase or rent) assistance 
for: 

Teachers 

_b 

Yes (TL-4) 

Principals 

_b 

Yes (TL-4) 

Financial incentives targeted toward 
increasing the number of staff with 
English language learner expertise in 
SIG schools for: 

Teachers 

11.9 

Yes (TL-4) 

Principals 

_b 

Yes (TL-4) 

Other financial incentives for: 
Teachers 

25.4 

No 

Principals 

11.9 

No 

Number of Districts 

60 



Source: Interviews with district administrators in spring 2012. 

a Districts that answered no to a question about whether the district offered any financial incentives to 
help recruit or retain effective teachers and/or principals are included in the analysis of this question as 
“no” responses. 

b This cell has been suppressed to protect respondent confidentiality. 

TL-4 = Implementing strategies to recruit, place, and retain staff. 
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Table C.20. District Use of Nonfinancial Strategies to Recruit or Retain Effective Staff in SIG 
Schools Implementing One of the SIG-Funded Intervention Models, Spring 2012 



Percentage of Districts 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported that principals had discretion to decide 
which staff to hire 

81.4 

Yes (TL-4) 

Reported modifying teacher tenure rules that 
affect placement and/or removal 

20.3 

Yes (TL-4) 

Reported using retention or recruitment efforts 
targeted toward increasing the number of staff 
with English language learner expertise 

39.0 

Yes (TL-4) 

Reported increasing the amount of induction 
support for novice teachers (above and beyond 
that provided to all novice teachers in the district) 
with the goal of increasing retention 

46.6 

Yes (TL-4) 

Other strategies 

22.0 

No 

Number of Districts 

60 



Source: Interviews with district administrators in spring 2012. 

TL-4 = Implementing strategies to recruit, place, and retain staff. 
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Table C.21. School-Reported Opportunities for Staff to Receive Financial Incentives, Spring 2012 



Percentage of Low-Performing Schools 

Implementing Not Implementing 

a SIG-Funded a SIG-Funded 

Intervention Model Intervention Model 
in 2011-2012 in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported offering signing or recruitment 
bonuses for: 

Teachers 

18.1 

10.4 

Yes (TL-4) 

Principals 

10.3 

7.0 

Yes (TL-4) 

Reported offering retention bonuses for: 
Teachers 

10.1 

6.1 

Yes (TL-4) 

Principals 

8.0 

2.5 

Yes (TL-4) 

Reported offering performance bonuses 
for: 

Teachers 

40.2 

32.9 

Yes (TL-4) 

Principals 

40.8 

30.6 

Yes (TL-4) 

Reported increasing annual 
compensation other than bonuses for: 
Teachers 

33.3 

23.6 

Yes (TL-4) 

Principals 

28.6 

19.0 

Yes (TL-4) 

Reported offering loan forgiveness for: 
Teachers 

52.7 

52.1 

Yes (TL-4) 

Principals 

16.3 

11.7 

Yes (TL-4) 

Reported offering tuition reimbursement 
for: 

Teachers 

40.9 

34.8 

Yes (TL-4) 

Principals 

30.1 

19.5 

Yes (TL-4) 

Reported offering housing (purchase or 
rent) assistance for: 

Teachers 

9.9 

10.5 

Yes (TL-4) 

Principals 

7.8 

4.5 

Yes (TL-4) 

Reported offering financial incentives 
targeted toward increasing the number of 
staff with English language learner 
expertise in the school for: 

Teachers 

6.0 

8.0 

Yes (TL-4) 

Principals 

2.3 

4.1 

Yes (TL-4) 

Reported offering other financial 
incentives for: 

Teachers 

16.2 

3.4 

Yes (TL-4) 

Principals 

7.8 

0.0 

Yes (TL-4) 

Number of Schools 

180-280 

110-170 



Source: Surveys of school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


TL-4 = Implementing strategies to recruit, place, and retain staff. 


C.23 




Appendix C. Detailed Findings From Interviews and Surveys 


Usage of Practices Promoted by SIG 


Table C.22. School-Reported Use of Nonfinancial Strategies to Recruit and Retain Staff, Spring 
2012 



Percentage of Low-Performing 
Schools 

Implementing Not Implementing 

a SIG-Funded a SIG-Funded 

Intervention Model Intervention Model 
in 2011-2012 in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported that principal had discretion to 
decide which staff to hire 

64.3 

69.8 

Yes (TL-4) 

Reported offering increased induction 
support for novice teachers in the school 
(above and beyond that provided to all 
novice teachers in the district) 

55.4 

51.5 

Yes (TL-3) 

Reported engaging in the following activities 
in 2011-2012: 

Provided additional professional 
development, mentoring, and/or 
instructional coaching to teachers and/or 
school leaders 3 

98.2 

94.0 

Yes (TL-3) 

Improved opportunities for collaboration 
(such as common planning time) 

95.4 

91.7 

Yes (TL-3) 

Improved the quality of school facilities 

75.3 

67.3 

No 

Increased availability of classroom or 
instructional supplies 

90.6 

79.9 

No 

Enhanced safety measures in the building 

79.9 

73.4 

Yes (TC-2) 

Increased access to technology for 
teachers 

94.6 

78.7 

Yes (IS-4) 

Offered more flexible work conditions (for 
example, flexible schedule) 

32.4 

26.8 

Yes (TL-4) 

Increased the use of 
aides/paraprofessionals 

52.3 

39.1 

Yes (TL-4) 

Increased the use of volunteers (for 
example, parents) 

64.0 

53.3 

Yes (TC-2) 

Other activities 

19.8 

9.9 

No 

Number of Schools 

120-280 

80-170 



Source: Surveys of school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Includes principals, assistant principals, or department heads. 

TL-3 = Providing high-quality, job-embedded professional development or supports; TL-4 = Implementing 
strategies to recruit, place, and retain staff; TC-2 = Engaging families and communities and providing a 
safe school environment that meets students’ social, emotional, and health needs; IS-4 = Using and 
integrating technology-based supports. 
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Table C.23. Factors Used by States to Select SIG Schools 



Among States That Did 
Not Fund All SIG-eligible 
Schools in 2010, 
Number of States 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported using the following factors to determine 
which persistently lowest-achieving schools 
would receive SIG funding: 

Funded schools in specific tiers 

13 

No 

Funded schools containing specific grade 
levels 

2 

No 

Funded schools with a high proportion of 
English language learners 

0 

No 

Funded schools with other specific student 
demographic characteristics 

1 

No 

Funded schools that were high poverty 

4 

No 

Funded schools that were committed to 
implementing one of the four SIG 
intervention models 

12 

No 

Funded schools located in districts that 
demonstrated capacity for reform 

10 

No 

Aimed to concentrate school funding in few 
districts 

1 

No 

Aimed to spread SIG funding across many 
districts 

8 

No 

Other factors 

3 

No 

Number of States 

15-16 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample size because nonresponse varied across items. The 

sample used for this analysis includes only the states in which the SIG-sample schools were 
located and that did not fund all SIG-eligible schools (the interview questions analyzed in this 
table were not asked of states that funded all SIG-eligible schools). 
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Table C.24. Factors Used by Districts to Select Schools to Include in Their SIG Applications 



Percentage of Districts 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported that the following were key 
considerations in selecting Tier 1 and Tier II 
schools: 



SIG eligibility tiers 

89.5 

No 

Grade level 

25.9 

No 

Percentage of English language learners 
in the school 

27.8 

No 

Schools with high poverty rates 

48.1 

No 

Other student demographic 
characteristics 

21.8 

No 

School commitment to implementing one 
of the four SIG intervention models 

75.0 

No 

School capacity for reform 

73.2 

No 

Previous academic achievement of the 
school 

82.1 

No 

Availability of funding from sources other 
than SIG 

41.8 

No 

Parent/community input 

60.7 

No 

School interest in participating 

54.5 

No 

School desire to retain current principal 

33.9 

No 

Existing, ongoing efforts to turn around 

69.6 

No 

Other factors 

40.7 

No 

Number of Districts 

50-60 



Source: Interviews with district administrators in spring 2012. 


Note: A range is provided for the sample size because nonresponse varied across items. 
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Table C.25. State Guidance to Districts on 
Schools 

Selecting a SIG Intervention 

Model for Individual 


Number of States 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported having provided the following 
types of guidance to districts regarding 
the selection of a school intervention 
model for each school: 

Allowed or prohibited specific models 
and/or strategies 

4 

No 

Guidance on how to match the model to 
school needs and capacity 

15 

No 

Guidance on models appropriate for 
addressing the needs of English 
language learners 

6 

No 

Guidance on how to engage the 
community in the selection of the 
model 

14 

No 

Some other type of guidance 

7 

No 

Number of States 

20-21 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample size because nonresponse varied across items. The 

sample used for this analysis includes only the states in which the SIG-sample schools were 
located. 
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Table C.26. Factors Considered by Districts When Selecting SIG Intervention Models for Individual 
Schools 



Among Districts That 
Reported Schools Currently 
Receiving SIG, Percentage 
of Districts 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported having considered the following 
factors when selecting the SIG intervention 
model to implement in these schools: 



State priorities and guidance 

78.0 

No 

Grade level of the school 

34.5 

No 

Previous academic achievement of the 
school 

86.4 

No 

Availability of funding from sources other 
than SIG 

42.1 

No 

Parent/community input 

69.0 

No 

School interest in and commitment to 
specific models 

60.3 

No 

Percentage of English language learners 
in the school 

26.3 

No 

Percentage of some other population of 
students in the school 

27.6 

No 

School desire to retain current principal 

41.4 

No 

Existing, ongoing efforts to turn around 
some of the eligible schools 

84.7 

No 

District and/or school capacity 

89.7 

No 

Other factors 

28.8 

No 

Number of Districts 

60 



Source: Interviews with district administrators in spring 2012. 
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Table C.27. Factors Considered by Schools When Selecting Their School Intervention Model 



Among Schools That Reported Using 
One of the Four SIG intervention 
Models, Percentage of 
Low-Performing Schools 



Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Not Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported considering the following 
factors when selecting an intervention 
model: 




State priorities and guidance 

56.5 

73.3 

No 

Previous academic achievement of 
the school 

94.6 

100.0 

No 

Availability of funding from sources 
other than SIG or RTT 

50.8 

51.7 

No 

Parent/community input 

59.9 

51.7 

No 

School interest in and commitment to 
specific models 

66.0 

55.2 

No 

The percentage of English language 
learners in the school a 

38.4 

55.0 

No 

The percentage of another population 
of students in the school 

20.8 

32.1 

No 

School desire to retain current 
principal 

28.0 

35.5 

No 

Existing, ongoing turnaround efforts 

66.8 

75.9 

No 

District and/or school capacity 

49.6 

44.4 

No 

Other factors 

3.1 

_b 

No 

Number of Schools 

160-260 

20-30 



Source: Surveys of school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Schools that reported that they did not have any English language learners are included in the analysis 
of this item as “no” responses. 

b This cell has been suppressed to protect respondent confidentiality. 
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Table C.28. Funds to Support School Improvement Efforts 


Percentage of Low-Performing Schools 



Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Not Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported receiving SIG in the 
following school years: 
2011-2012 

97. 5 a 

28.5 

No 

2010-2011 

89.9 

30.3 


Reported being in a state that 

received an RTT grant 

47.1 

38.2 

No 

Reported receiving RTT funds 
specifically for school improvement 
efforts in the following school years: b 
2011-2012 

28.9 

15.1 

No 

2010-2011 

24.8 

10.9 


Number of Schools 

270-280 

170 



Source: Surveys of school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


a This number is less than 100 percent because we used several sources of information (other than the 
survey of school administrators) to identify the set of schools that implemented a SIG-funded intervention 
model in 2011-2012, and some schools that were identified as being in this group reported on the survey 
that they did not receive SIG. 

b Schools that responded no to the question in the prior row are included in the analysis of this question 
as “no” responses. 
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Table C.29. District-Reported School Expenditures 



Schools 
Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Schools Not 
Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Mean total school expenditures 
in: 

2009-2010 

$6,125,000 

$6,977,000 

No 

2011-2012 

$6,645,000 

$6,228,000 


Mean percentage of school 
expenditures that go to wages, 
employee benefits, and other 
personnel expenditures in: 
2009-2010 

86.6 

83.6 

No 

2011-2012 

84.9 

83.7 


Number of Schools 

270-280 

170 



Source: Interviews with district administrators in spring 2012. District-reported values for individual 

schools were ascribed to the appropriate schools in the study sample. 

Note: A range is provided for the sample sizes because nonresponse varied across items. School 

expenditures were rounded to the nearest thousand. 
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Table C.30. School Intervention Models Used in Study Schools, Spring 2012 



Percentage of Low-Performing Schools 

Implementing Not Implementing 

a SIG-Funded a SIG-Funded 

Intervention Model Intervention Model 
in 2011-2012 in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported using one of the four SIG 
intervention models 

96.0 

18.1 

No 

Reported using the following SIG 
intervention models: b 
Turnaround model (replace the 
principal and rehire no more than 
50 percent of staff; increase 
operational flexibility and learning 
time; make changes to the 
instructional program and 
professional development) 

44.4 

9.0 

No 

Transformation model (implement 
changes similar to those specified 
for the turnaround model, except [1] 
there are no limits on rehiring staff 
and [2] student growth must factor 
into teacher and principal 
evaluations) 

46.2 

_d 

No 

Restart model (close the school and 
reopen under a charter or 
education management 
organization) 

5.4 

_d 

No 

Closure model (close the school and 
send current students to higher- 
achieving schools in the district) 

0.0 

_d 

No 

Transformation, restart, or closure 
model 

_d 

9.0 


Reported being a charter school 

7.2 

3.0 

No 

Number of Schools 

280 

170 



Source: Surveys of school administrators in spring 2012. 


a This number is less than 100 percent because we used several sources of information (other than the survey of 
school administrators) to identify the set of schools that implemented a SIG-funded intervention model in 2011-2012, 
and some schools that were identified as being in this group reported on the survey that they did not use one of the 
four SIG intervention models. 

b Schools that responded no to the question in the prior row are included in the analysis of this question as “no” 
responses. 

c This cell has been suppressed to protect respondent confidentiality. 

d To comply with NCES statistical reporting requirements for small cell sizes, we aggregated the percentages for the 
transformation, restart, and closure models for schools not implementing a SIG-funded intervention model. 
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Table C.31. Improvement Strategies Used in Study Schools, Spring 2012 



Percentage of Low-Performing Schools 

Implementing Not Implementing 

a SIG-Funded a SIG-Funded 

Intervention Model Intervention Model 
in 2011-2012 in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported implementing changes to the 
following since July 2010: 

English language arts curriculum 

70.2 

60.5 

No 

Math curriculum 

72.4 

63.0 

No 

Instructional approaches in English 
language arts 

92.0 

77.4 

No 

Instructional approaches in math 

91.9 

77.7 

No 

Strategies to meet the needs of 
English language learners 3 

60.8 

61.4 

Yes (IS-3) 

School administrative structure 

85.3 

55.1 

No 

Discipline policies 

85.8 

66.7 

Yes (TC-2) 

Nonacademic supports for students 

81.4 

63.3 

Yes (TC-2) 

Policies or strategies related to 
parent and/or community 
engagement 

90.8 

68.3 

Yes (TC-2) 

Policies around the use of data for 
instructional improvement 

91.5 

77.8 

Yes (IS-2) 

Monitoring of student readiness for 
grade promotion and/or high 
school graduation 

78.4 

68.1 

Yes (IS-5) 

Monitoring of students’ college 
readiness b 

89.8 

87.3 

Yes (IS-5) 

Other changes 

6.3 

9.7 

No 

Number of Schools 

130-280 

80-170 



Source: Surveys of school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Schools that reported that they did not have any English language learners are included in the analysis 
of this item as “no” responses. 
b The analysis for this row includes only high schools. 

IS-2 = Promoting the continuous use of student data; IS-3 = Providing supports and professional 
development to staff to assist ELLs and students with disabilities; IS-5 = Tailoring strategies for secondary 
schools; TC-2 = Engaging families and communities and providing a safe school environment that meets 
students’ social, emotional, and health needs. 
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Table C.32. Instructional Strategies Used to Meet the Needs of English Language Learners, 
Spring 2012 



Among Schools That Reported Having 
English Language Learners, Percentage 
of Low-Performing Schools 

Implementing Not Implementing 

a SIG-Funded a SIG-Funded 

Intervention Model Intervention Model 
in 2011-2012 in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported using the following 
strategies to meet the needs of these 
students: 

Used a curriculum that specifically 
addresses ELL needs 

77.6 

66.1 

Yes (IS-3) 

Implemented instructional 
strategies that specifically 
address ELL needs 

86.9 

84.0 

Yes (IS-3) 

Provided instructional programs 
specifically designed for ELL 

80.3 

80.0 

Yes (IS-3) 

Provided specialized classes for 

ELL 

70.8 

68.3 

Yes (IS-3) 

Provided additional services for 
ELL 

79.5 

77.4 

Yes (IS-3) 

Provided professional development 
for teachers on providing 
instruction to ELL 

79.8 

75.2 

Yes (IS-3) 

Used data on ELL in school 
decision making 

85.6 

87.2 

Yes (IS-3) 

Other strategies 

4.3 

2.6 

No 

Number of Schools 

160-180 

120-130 



Source: Surveys of school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

ELL = English language learner; IS-3 = Providing supports and professional development to staff to assist 
ELLs and students with disabilities. 
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Table C.33. District Administrative Supports for Turnaround 



Percentage of Districts 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported having the following 
supports in place related to school 
turnaround: 

Staff explicitly designated to support 
school turnaround (but no 
designated turnaround office) 


Yes (FS-2) 

2009-2010 

44.0 


2011-2012 

60.0 


An office explicitly designated to 
support school turnaround (with 
designated staff) 


Yes (FS-2) 

2009-2010 

16.0 


2011-2012 

36.0 


Contracts with external consultants 
to support school turnaround 


Yes (FS-2) 

2009-2010 

55.9 


2011-2012 

78.0 


Other supports 


No 

2009-2010 

15.3 


2011-2012 

27.1 


Number of Districts 

50-60 



Source: Interviews with district administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


FS-2 = Receiving technical assistance and support. 
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Table C.34. Flexibility With or Exemptions From Collective Bargaining Agreements or Staffing 
Policies for SIG Schools Implementing One of the Four SIG Intervention Models 



Percentage of Districts 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported that SIG grantee schools had 
flexibility from the following aspects of 
collective bargaining agreements or policies 
that guide staffing in other district schools: 
Procedures for assigning or removing staff 

in: 


Yes (FS-1) 

2009-2010 

31.3 


2011-2012 

62.5 


Requirements or policies related to staff 

hours and responsibilities in: 


Yes (FS-1) 

2009-2010 

22.9 


2011-2012 

66.7 


Procedures related to the distribution of 

effective staff in: 


Yes (FS-1) 

2009-2010 

12.5 


2011-2012 

41.7 


Other types of flexibility or exemptions: 


No 

2009-2010 

10.4 


2011-2012 

35.4 


Number of Districts 

50 



Source: Interviews with district administrators in spring 2012. 


FS-1 = Having operational flexibility. 
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Table C.35. School Responsibility for Decision Making, Spring 2012 


- 

Percentage of Low-Performing Schools 

Implementing Not Implementing 

a SIG-Funded a SIG-Funded 

Intervention Model Intervention Model 

in 2011-2012 in 201 1-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported having primary 
responsibility for making 
decisions in the following areas 
(rather than the state or 
district): 

Setting student discipline 
policies 

38.8 

33.1 

Yes (FS-1) 

Developing the school 
budget 

55.4 

53.6 

Yes (FS-1) 

Establishing the curriculum 
(including core texts) 

18.0 

15.7 

Yes (FS-1) 

Setting student assessment 
policies (on assessments 
other than state-mandated 
tests) 

24.7 

21.7 

Yes (FS-1) 

Staff hiring, discipline, and 
dismissal 

37.4 

45.8 

Yes (FS-1) 

Determining the length of the 
school day 

18.1 

12.0 

Yes (FS-1) 

Determining the length of the 
school year 

6.6 

3.6 

Yes (FS-1) 

Setting requirements for 
professional development 

52.2 

40.4 

Yes (FS-1) 

Number of Schools 

270 

170 



Source: Surveys of school administrators in spring 2012. 


FS-1 = Having operational flexibility. 
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Table C.36. Organization of Instruction in Schools, Spring 2012 



Percentage of Low-Performing Schools 

Implementing Not Implementing 

a SIG-Funded a SIG-Funded 

Intervention Model Intervention Model 

in 2011-2012 in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported using the following methods 
to organize classes or other groups of 
students for instruction: 

Traditional grades or academic 
discipline-based departments 

84.6 

87.3 

No 

Grades or the school subdivided into 
small learning communities 

56.5 

56.4 

Yes (IS-5) 

Student groups that remain two or 
more years with the same teacher 

29.0 

33.5 

No 

Interdisciplinary teaching or 
paired/team teaching 

57.1 

47.6 

No 

Specialized classes for ELL a 

54.6 

57.2 

Yes (IS-3) 

Block scheduling 

49.3 

56.1 

Yes (TC-1) 

Other methods 

5.7 

6.8 

No 

Number of Schools 

270 

160-170 



Source: Surveys of school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Schools that reported that they did not have any English language learners are included in the analysis 
of this item as “no” responses. 

ELL = English language learner; IS-5 = Tailoring strategies for secondary schools; IS-3 = Providing 
supports and professional development to staff to assist ELLs and students with disabilities; TC-1 = 
Increasing learning time. 
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Table C.37. School Instructional Time 



Percentage of Low-Performing Schools 



Schools 

Schools Not 



Implementing 

Implementing 

Item Aligned with 


a SIG-Funded 

a SIG-Funded 

SIG Application 


Intervention Model 

Intervention Model 

Criteria 


in 2011-2012 

in 2011-2012 

(Subtopic) 

Mean number of instructional days that 
schools report being in session for 
students in: 



Yes (TC-1) 

2009-2010 

181 

179 


2011-2012 

182 

179 


Mean number of hours per day that 
schools report being in session for 
students in: 



Yes (TC-1) 

2009-2010 

6.9 

6.8 


2011-2012 

7.1 

6.8 


Mean number of minutes per week of 
instruction that schools report providing 
to the average student in: 




Mathematics 

361 

328 

No 

English language arts 

379 

362 

No 

Number of Schools 

220-260 

140-160 



Source: Surveys of school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

TC-1 = Increasing learning time. 
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Table C.38. School’s Offerings Outside the Regular School Day, Spring 2012 



Percentage of Low-Performing Schools 



Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Not Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported using or offering the 
following: 

Before- and/or after-school 
instruction 

89.9 

77.2 

Yes (TC-1) 

Weekend instruction 

45.4 

43.2 


Summer instruction 

75.8 

57.9 


Number of Schools 

270-280 

160-170 



Source: Surveys of school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

TC-1 = Increasing learning time. 
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Table C.39. Common Planning Time, Spring 2012 



Percentage of Low-Performing Schools 



Implementing 

Not Implementing 

Item Aligned with 


a SIG-Funded 

a SIG-Funded 

SIG Application 


Intervention Model 

Intervention Model 

Criteria 


in 2011-2012 

in 2011-2012 

(Subtopic) 

Reported that all or some teachers 
have common planning time to 
meet in teams 

93.8 

94.6 

Yes (TL-3) 

Reported that all or some teachers 
have common planning time with 
the following frequency: 3 




Daily 

46.4 

40.5 

No 

Several times per week 

22.1 

24.4 

No 

Once per week 

22.8 

26.2 

No 

Monthly, or a few times per year b 

2.5 

3.6 

No 

Reported that all or some teachers 
are required to participate in 
common planning time 3 

85.5 

82.9 

No 

Number of Schools 

270-280 

160-170 



Source: Surveys of school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Schools that responded no to the question about whether teachers have common planning time to meet 
in teams are included in the analysis of this question as “no” responses. 

b To comply with NCES statistical reporting requirements for small cell sizes, we aggregated the 
percentages for “monthly” and “a few times per year.” 

TL-3 = Providing high-quality, job-embedded professional development or supports. 
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Table C.40. Frequency of Use of Benchmark Tests in English Language Arts and Math, 
Spring 2012 



Percentage of Low-Performing 



Schools 



Implementing 

Not Implementing 



a SIG-Funded 

a SIG-Funded 

Item Aligned with 


Intervention 

Intervention 

SIG Application 


Model in 2011— 

Model in 2011— 

Criteria 


2012 

2012 

(Subtopic) 

Reported that the typical English 
language arts teacher uses 
benchmark or interim assessments 
with the following frequency: 



Yes (IS-2) 

Zero to two times per year 3 

5.5 

12.0 


Three to four times per year 

54.9 

49.7 


Five to six times per year 

17.2 

17.4 


Seven to eight times per year 

6.6 

9.6 


More than eight times per year 

15.8 

11.4 


Reported that the typical math teacher 
uses benchmark or interim 




assessments with the following 
frequency: 



Yes (IS-2) 

Zero to two times per year 3 

5.9 

12.6 


Three to four times per year 

51.7 

47.3 


Five to six times per year 

18.5 

18.6 


Seven to eight times per year 

8.5 

10.2 


More than eight times per year 

15.5 

11.4 


Number of Schools 

270 

170 



Source: Surveys of school administrators in spring 2012. 

a To comply with NCES statistical reporting requirements for small cell sizes, we aggregated the 
percentages for “zero times per year” and “one to two times per year.” 

IS-2 = Promoting the continuous use of student data. 
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Table C.41. Changes in Staff Implemented as Part of School Improvement Efforts 



Percentage of Low-Performing Schools 



Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Not Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported getting a new principal 
between July 2010 and spring 2012 

69.2 

49.4 

Yes (TL-2) 

Reported pursuing other significant 
leadership changes (aside from the 
principal) between July 2010 and spring 
2012 

52.5 

27.4 

No 

Reported having removed instructional 
staff through firing or counseling out 
between July 2010 and spring 2012 

57.8 

40.8 

Yes (TL-2) 

Among schools that reported having 
removed instructional staff through firing 
or counseling out between July 2010 
and spring 2012, average proportion of 
existing instructional staff that was 
removed 

29.9 

21.7 

No 

Reported having hired a significant 
number of new staff (at least 50 percent 
of staff or more) between July 2010 and 
spring 2012 

42.2 

13.1 

Yes (TL-2) 

Reported having reviewed the strengths 
and competencies of all existing 
instructional staff to assess the extent to 
which they were likely to be successful 
working in a school turnaround or 
improvement context between July 
2010 and spring 2012 

68.8 

45.0 

Yes (TL-2) 

Reported having assessed new hires for 
whether they possessed specific 
strengths or competencies deemed 
important to be successful working in a 
school turnaround or improvement 
context between July 2010 and spring 
2012 

31.4 

9.5 

Yes (TL-2) 

Number of Schools 

160-280 

70-170 



Source: Surveys of school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

TL-2 = Identifying and rewarding effective teachers and principals and removing ineffective ones. 
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Table C.42. School-Reported Training or Technical Assistance from the State or District 



Percentage of Low-Performing Schools 



Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Not Implementing 
a SIG-Funded 
Intervention Model 
in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported that the state and/or district 
provided the following types of training 
or technical assistance to the school 
since July 2010: 




Training or technical assistance on 
developing and implementing a 
school improvement plan 

84.1 

76.8 

Yes (FS-2) 

Training or technical assistance on 
identifying curricula, instructional 
strategies, or school reform 
models that have been shown to 
be effective at increasing student 
achievement 

79.9 

75.0 

Yes (FS-2) 

Training or technical assistance on 
identifying curricula, instructional 
strategies, or school reform 
models that have been shown to 
be effective at improving college 
readiness 

64.0 

61.6 

Yes (FS-2) 

Training or technical assistance on 
developing strategies to recruit 
and retain more effective teachers 

46.3 

35.4 

Yes (FS-2) 

Other assistance 

5.7 

4.9 

No 

Number of Schools 

270 

160 



Source: Surveys of school administrators in spring 2012. 

FS-2 = Receiving technical assistance and support. 


C.44 





Appendix C. Detailed Findings From Interviews and Surveys 


Usage of Practices Promoted by SIG 


Table C.43. Professional Development for School Instructional Staff, Spring 2012 



Percentage of Low-Performing Schools 

Implementing Not Implementing 

a SIG-Funded a SIG-Funded 

Intervention Model Intervention Model 
in 2011-2012 in 2011-2012 

Item Aligned with 
SIG Application 
Criteria 
(Subtopic) 

Reported that their instructional staff 
received professional development on 
the following topics: 

Transitioning to the CCSS 

71.3 

70.1 

Yes (TL-3) 

Aligning instruction to state standards 

86.1 

77.6 

Yes (TL-3) 

Instructional strategies 

94.3 

80.5 

Yes (TL-3) 

Using data to improve and/or 
differentiate instruction 

90.1 

82.1 

Yes (TL-3) 

Meeting the needs of English 
language learners 3 

53.3 

54.2 

Yes (IS-3) 

Strategies for turning around a low- 
performing school 

74.8 

42.2 

Yes (TL-3) 

Other topics 

5.9 

8.9 

No 

Reported that the following 
characteristics apply to at least half of 
the professional development activities 
provided to instructional staff: 
Single-session, one-time events 

31.6 

40.2 

No 

Multiple-session events 

84.7 

75.3 

Yes (TL-3) 

Involved practice in the classroom 

84.8 

77.9 

Yes (TL-3) 

Required for all instructional staff 

94.2 

95.2 

No 

Were designed with input from school 
staff 

82.2 

70.5 

Yes (TL-3) 

Number of Schools 

260-280 

160-170 



Source: Surveys of school administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Schools that reported that they did not have any English language learners are included in the analysis 
of this item as “no” responses. 

CCSS = Common Core State Standards; TL-3 = Providing high-quality, job-embedded professional 
development or supports; IS-3 = Providing supports and professional development to staff to assist ELLs 
and students with disabilities. 
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Appendix D. Survey Questions Aligned with SIG Practices 


Usage of Practices Promoted by SIG 


This appendix provides a crosswalk between each practice aligned with the SIG application 
criteria and the school administrator survey questions. For each of the four SIG topic areas 
addressed in Chapter IV, this appendix presents a table showing the survey questions that 
address the area practices aligned with the SIG application criteria. The school administrator 
survey protocol is available at http://www.mathematica-mpr.com/publications/PDFs 
/Spring_20 12_School_Administrator_Survey.pdf. 


Table D.l. Survey Questions Addressing the Comprehensive Instructional Reform Strategies 
Topic Area Practices 


Survey Questions Addressing the Comprehensive Instructional 
Practice Reform Strategies Practice 

Using data to evaluate DAI . During the current school year, for which of the following purposes has 
instructional programs your school used data? a. To evaluate instructional programs (for example, 

(for example, measuring program effectiveness) 

measuring program 

effectiveness) 

Using data to inform DAI . During the current school year, for which of the following purposes has 

and differentiate your school used data? b. To guide development and implementation of 

instruction academic supports or enrichment programs (for example, identify how many 

and which students need academic support or enrichment, assign or reassign 
students to classes); d. To inform teachers’ instructional practices (for example, 
identify areas for improvement, tailor instruction to meet student needs, 
manage instructional pacing); g. To track individual student performance and 
identify areas of improvement for specific students; k. To inform resource 
allocation to improve instruction (for example, which students participate in 
which programs, which staff work with which students) 

DA2. During the current school year, for which of the following purposes has 
your school used data on English language learners? e. To 
inform/improve/differentiate instruction for English language learners 

DA3. Within the past year, did any of the following activities related to data use 
occur in your school? If so, how often did they occur (daily, weekly, monthly, a 
few times per year, or once per year)? For item b below, if your school does not 
have English language learners, select — NA. a. District staff met with you 
and/or other school staff to review data on overall student performance; b. 
District staff met with you and/or other school staff specifically to review student 
performance data on English language learners; c. You or other school leaders 
reviewed student performance data to identify areas of improvement for the 
school; d. You or other school leaders met with teachers to discuss student 
performance data to identify areas in need of improvement for individual 
students or groups of students; h. After reviewing student performance data, 
teachers, administrators, and/or coaches formulated specific plans to update 
and revise instructional practice to address issues with specific students or 
specific classes. 


The typical 

English/language arts 
or math teacher used 
benchmark or interim 
assessments at least 
once per year 


TA12. Since July 2010, did your school implement changes to any of the 
following? j. Policies around the use of data for instructional improvement 
TA37. How often does the typical English language arts teacher in your school 
use benchmark or interim assessments? 

TA38. How often does the typical math teacher in your school use benchmark 
or interim assessments? 
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Table D.1 (continued) 


Practice 

Implementing 
strategies (including 
additional supports or 
PD) to ensure that 
limited English 
proficient students 
acquire language skills 
to master academic 
content 


Survey Questions Addressing the Comprehensive Instructional 
Reform Strategies Practice 

TL29. During the current school year, have the state and/or district provided 
professional development or other support to the principal and/or other leaders 
of this school on any of the following topics? e. Ensuring that English language 
learners acquire the language skills needed to master academic content 

DA2. During the current school year, for which of the following purposes has 
your school used data on English language learners? If your school does not 
have English language learners, select — NA. a. To make decisions about 
students’ entry into and/or exit from English language learner status; b. To 
place English language learners into specialized programs and/or classes; c. 
To track the progress of current English language learners; d. To track the 
progress of former English language learners; f. To identify professional 
development needs for teachers of English language learners; g. To assess 
teacher effectiveness with English language learners 


DA3. Within the past year, did any of the following activities related to data use 
occur in your school? If so, how often did they occur (daily, weekly, monthly, a 
few times per year, or once per year)? For item f below, if your school does not 
have English language learners, select — NA. f. School leaders coached 
teachers on the use of data specifically to improve instruction of English 
language learners. 


DAIO.This school year, has your school received any of the following supports 
to help your school access and use data related to English language learners to 
improve and/or differentiate instruction for these students? For each type of 
support received, please describe the nature of the support received. If your 
school does not have English language learners, select — NA. a. Supports to 
help school staff use data to track the performance of English language 
learners (Please specify); b. Supports to help school staff use data to improve 
or differentiate instruction for English language learners (Please specify) 


TA12. Since July 2010, did your school implement changes to any of the 
following? For item e below, if your school does not have English language 
learners, select — NA. e. Strategies to meet the needs of English language 
learners 


TA22. Which of the following topics have been a focus of the professional 
development provided to instructional staff this school year? For item e below, if 
your school does not have English language learners, select — NA. e. Meeting 
the needs of English language learners 

TA32. Is your school currently using any of the following methods to organize 
classes or other groups of students for instruction? For item e below, if your 
school does not have English language learners, select — NA. e. Specialized 
classes for English language learners (such as newcomer class, English as a 
second language, sheltered content) 

TA36. Which of the following strategies/approaches does your school currently 
use to meet the needs of your school’s English language learners? a. Use a 
curriculum that specifically addresses English language learners needs (Please 
specify); b. Implement instructional strategies that specifically address English 
language learners’ needs, such as needs-based grouping, differentiated 
instruction, or increased progress testing of English language learners (Please 
specify); c. Provide instruction programs specifically designed for English 
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Table D.1 (continued) 


Practice 

Survey Questions Addressing the Comprehensive Instructional 
Reform Strategies Practice 

language learners (such as English as a second language or bilingual 
programs) (Please specify); d. Provide specialized classes for English language 
learners (such as newcomer class, sheltered content class) (Please specify); 
e. Provide additional services for English language learners (such as tutors, 
bilingual aides, after-school program) (Please specify); f. Provide professional 
development for teachers on providing instruction to English language learners; 
g. Use data on English language learners in school decision making 

Increased access to 
technology for teachers 
or that the typical 
English/language arts 
teacher used 
computer-assisted 
instruction 

TA31 . This school year, how often does the typical English language arts 
teacher in your school engage in the following activities? d. Use computer- 
assisted instruction 

TL28. Within the past year, has your school engaged in any of the following 
activities? f. Increased access to technology for teachers 

Secondary school 
monitored students’ 
college readiness 
(such as enrollment in 
Advanced Placement 
courses), including 
providing supports 
(such as project-based 
learning) so that low- 
achieving students can 
take advantage of 
these types of 
opportunities 

DAI . During the current school year, for which of the following purposes has 
your school used data? i. To track preparation for college enrollment (for 
example, participation in Advanced Placement courses or dual enrollment) 

TA12. Since July 2010, did your school implement changes to any of the 
following? 1. Monitoring of students' college readiness (for example, 
participation in Advanced Placement courses, dual enrollment) 

TA31 . This school year, how often does the typical English language arts 
teacher in your school engage in the following activities? a. Use project-based 
learning (for example, hands-on, inquiry-based activities) in classes; c. Use 
tiered interventions (for example, targeted/pull-out services for struggling 
students, intensive support to students who do not respond to interventions) 

The school or grades 
within the secondary 
school were subdivided 
into small learning 
communities or 
field/career-oriented 
academies 

TA32. Is your school currently using any of the following methods to organize 
classes or other groups of students for instruction? b. Grades or the school 
subdivided into small learning communities, such as "houses," "families," 
"teams," or field/career-oriented "academies" such as health or sciences 

Secondary school 
tracked student 
progress towards (and 
readiness for) high 
school graduation 

DAI . During the current school year, for which of the following purposes has 
your school used data? h. To track student progress toward high school 
graduation (for example, credits earned, required courses taken) 

TA12. Since July 2010, did your school implement changes to any of the 
following? k. Monitoring of student readiness for grade promotion and/or high 
school graduation 


Source: Surveys of school administrators in spring 2012. 

Note: DA indicates that the question came from the data systems module of the survey. TA 

indicates that the question came from the school turnaround module of the survey. TL 
indicates that the question came from the teachers and leaders module of the survey. 
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Table D.2. Survey Questions Addressing the Teacher and Principal Effectiveness Topic Area 
Practices 


Practice 

Survey Questions Addressing the Teacher and 
Principal Effectiveness Practice 

Student achievement 
growth was a required 
component of teacher 
evaluations and the 
extent to which student 
achievement growth must 
factor into teacher 
evaluations or that state 
test scores were used to 
assess student growth for 
teacher evaluations was 
specified 

TL2. Currently, to what extent does student growth evidence factor into the 
overall teacher evaluation? For example, student growth may be a 
"significant" factor in evaluations or have a specific weight (such as 20 
percent) in the overall teacher evaluation. If this varies for different types of 
teachers, please describe this variation. 

TL3. Are any of the following measures used to assess student growth for 
teacher evaluations? a. State test scores; b. Scores on standardized 
assessments other than state tests; c. Some other measure of achievement 
(Please specify) 

[Note: TL1 (shown in the next row) was also used to address the practice in 
this row. Specifically, the practice in this row was coded as 0 if, among other 
things, the response to TL1 was “no teachers.”] 

Using multiple 
performance measures 
for teacher evaluations 

TL1 . Currently, are measures of student growth a required component of 
teacher evaluations? 

TL8. Apart from the student growth measures just addressed, which of the 
following other measures of teacher performance are currently used by your 
school for teacher evaluations? If a particular measure is used only for some 
teachers, please specify the types of teachers for whom the measure is used, 
a. Classroom observations conducted by the principal; b. Classroom 
observations conducted by someone other than the principal (such as a peer 
or mentor teacher); c. Self-assessment; d. Peer assessments; e. Portfolios or 
other artifacts of teacher practice; f. Student work samples; g. Student 
surveys or other feedback; h. Parent surveys or other feedback 

Using teacher evaluation 
results to inform decisions 
about compensation 

TL14. Currently, do teacher evaluation results contribute to decisions about 
annual salary increases for teachers in your school? 

TL16. Currently, do teacher evaluation results contribute to the decision to 
provide bonuses or other performance-based compensation (other than 
annual salary increases) for teachers in your school? 

Reviewing the strengths 
and competencies of 
instructional staff for the 
purposes of hiring or 
removing staff 

TA16. Since July 2010, did your school review the strengths and 
competencies of all existing instructional staff to assess the extent to which 
they were likely to be successful working in a school turnaround or 
improvement context? 

TA18. Since July 2010, did your school remove instructional staff through 
firing or counseling out as part of school improvement efforts? 


TA20. Since July 2010, did your school hire a significant number of new staff 
(at least 50 percent of staff or more) as part of school improvement efforts? 


TA21 . Were these new hires assessed for whether they possessed specific 
strengths or competencies deemed important to be successful working in a 
school turnaround or improvement context? 

Providing instructional 
staff with PD that 
consisted mostly or 
entirely of multiple- 
session events 

TA23. How would you characterize the nature of the professional 
development activities provided to instructional staff in your school this year in 
terms of the following characteristics? For example, focusing on the first row 
below, would you say that all, most, roughly half, few, or none of the 
professional development provided to instructional staff this school year were 
single-session, one-time events? b. Multiple-session events 
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Table D.2 (continued) 


Practice 

Providing instructional 
staff with PD that focused 
on transitioning to 
Common Core State 
Standards, aligning 
instruction to state 
standards, or strategies 
for turning around a low- 

performing school 

Providing staff with PD 
that involved educators 
working collaboratively or 
was facilitated by school 
leaders or coaches 


Providing staff with PD 
that was focused on 
understanding and 
addressing student 
learning needs (including 
reviewing student work 
and achievement data 
and collaboratively 
planning, testing, and 
adjusting instructional 
strategies based on data) 


Survey Questions Addressing the Teacher and 
Principal Effectiveness Practice 

TA22. Which of the following topics have been a focus of the professional 
development provided to instructional staff this school year? a. Transitioning 
to the Common Core State Standards; b. Aligning instruction to state 
standards; f. Strategies for turning around a low-performing school (Please 
specify) 


TL27. Currently, does your school offer increased induction support (above 
and beyond that provided to all novice teachers in the district) for novice 
teachers in this school? 

TL28. Within the past year, has your school engaged in any of the following 
activities? a. Provided additional professional development, mentoring, and/or 
instructional coaching to teachers and/or school leaders (such as principals, 
assistant principals, or department heads); b. Improved opportunities for 
collaboration such as common planning time 

DA3. Within the past year, did any of the following activities related to data 
use occur in your school? If so, how often did they occur (daily, weekly, 
monthly, a few times per year, or once per year)? e. School leaders coached 
teachers on the use of data to improve instruction; g. Teachers met with each 
other to discuss data on their students/classes. 

DA6. Does your school provide scheduled time for teachers to examine data, 
either on their own or in collaboration with other teachers or school 
administrators? 

TA33. Currently, do all, some, or no teachers in your school have common 
planning time to meet in teams? If some (but not all) teachers have common 
planning time, please specify which teachers have common planning time. 

TA23. How would you characterize the nature of the professional 
development activities provided to instructional staff in your school this year in 
terms of the following characteristics? For example, focusing on the first row 
below, would you say that all, most, roughly half, few, or none of the 
professional development provided to instructional staff this school year were 

single-session, one-time events? c. Involved practice in the classroom 

DA9. This school year, has your school received any professional 
development, training, or technical assistance to help school administrators 
and/or teachers access data, navigate data systems, or interpret and use 
data to improve and/or differentiate instruction? If so, please indicate the total 
number of hours of professional development, training, or technical 
assistance provided to school administrators and/or teachers this school year 
on these topics. 

DAI . During the current school year, for which of the following purposes has 
your school used data? e. To inform professional development offerings (for 
example, identify specific content or skills in which teachers need assistance 
or support) 
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Table D.2 (continued) 


Practice 

Survey Questions Addressing the Teacher and 
Principal Effectiveness Practice 


TL10. Currently, are teacher evaluation results used to guide decisions about 
what professional development and support is offered, recommended, or 
required for individual teachers in your school? 


TA22. Which of the following topics have been a focus of the professional 
development provided to instructional staff this school year? c. Instructional 
strategies (Please specify which instructional strategies were part of the 
professional development); d. Using data to improve and/or differentiate 
instruction (Please specify the specific strategies to improve and/or 
differentiate instruction that were part of the professional development) 

Providing staff with PD 
designed with input from 
school staff 

TA23. How would you characterize the nature of the professional 
development activities provided to instructional staff in your school this year in 
terms of the following characteristics? e. Were designed with input from 
school staff 

Using data to evaluate 
the success of PD 
offerings 

DAI . During the current school year, for which of the following purposes has 
your school used data? f. To evaluate the success of professional 
development offerings 

Implementing strategies, 
such as financial 
incentives or more flexible 
work conditions, that were 
designed to recruit, place, 
and retain staff 

TL18. Currently, are teacher evaluation results used to guide decisions about 
career advancement for teachers in your school? 

TL26. Currently, do teachers and/or the principal at your school have the 
opportunity to receive any of the following financial incentives? a. 
Signing/recruitment bonuses for beginning to work in this school; b. Retention 
bonuses for continuing to work in the school; c. Performance bonuses; d. 
Increased annual compensation other than bonuses; e. Loan forgiveness; f. 
Tuition reimbursement; g. Housing; h. Financial incentives targeted towards 
increasing the number of staff with English language learner expertise in the 
school 


TL28. Within the past year, has your school engaged in any of the following 
activities? g. Offered more flexible work conditions (for example, more flexible 
schedule); h. Increased use of aides/paraprofessionals 

Using teacher evaluation 
results as the primary 
consideration in 
reductions in force and 
excessing decisions or 
having teacher 
assignment policies that 
allow for principal 
discretion to decide which 
staff to hire for the school 

TL13. Currently, are teacher evaluation results, rather than seniority, the 
primary consideration in reductions in force and excessing decisions for your 
school (if your school were to reduce the size of its faculty)? 

TL25. Do current teacher-assignment policies for your school allow for 
principal discretion or authority to decide which staff to hire for your school? If 
yes, please describe the discretion or authority available to your school’s 
principal when making hiring decisions. 

Measures of student 
achievement growth were 
used for principal 
evaluations and the 
extent to which student 
achievement growth must 
factor into principal 
evaluations was specified 

TL20. Currently, to what extent does student growth factor into the overall 
principal evaluation? For example, student growth may be a "significant" 
factor in evaluations or have a specific weight (such as 20 percent) in the 
overall principal evaluation. 

[Note: TL19a (shown in the next row) was also used to address the practice in 
this row. Specifically, the practice in this row was coded as 0 if, among other 
things, the response to TL19a was "no.”] 

Using multiple 
performance measures 
for principal evaluations 

TL19. Currently, which of the following measures are used to evaluate the 
performance of your school’s principal? a. Student growth measures; b. Self- 
assessment; c. District administrator input; d. School staff surveys or other 
feedback; e. Student surveys or other feedback 
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Table D.2 (continued) 


Practice 

Survey Questions Addressing the Teacher and 
Principal Effectiveness Practice 

Principal evaluation 
results were used to 
inform decisions about 
compensation 

TL22. Currently, do principal evaluation results contribute to decisions about 
annual salary increases for the principal of your school? 

TL24. Currently, do principal evaluation results contribute to the decision to 
provide bonuses or performance-based compensation to the principal of your 
school? 

School has a new 
principal 

TA1 3. Did your school get a new principal between July 201 0 and June 
2011? 


TAM. Did your school get a new principal between July 201 1 and today? 

State or district provides 
the principal or other 
school leaders with PD on 
analyzing and revising 
budgets or strategies for 
turning around a low- 
performing school 

TL29. During the current school year, have the state and/or district provided 
professional development or other support to the principal and/or other 
leaders of this school on any of the following topics? f. Analyzing and revising 
budgets to use resources effectively; g. Strategies for turning around a low- 
performing school 

State or district provides 
the principal or other 
school leaders with PD on 
identifying effective 
instructional staff for 
leadership positions and 
supporting them in these 
positions 

TL29. During the current school year, have the state and/or district provided 
professional development or other support to the principal and/or other 
leaders of this school on any of the following topics? b. Identifying effective 
instructional staff for leadership positions and supporting them in such 
positions 

State or district uses 
principal evaluation 
results to develop the 
principal’s PD or provides 
the principal with PD on 
aligning teachers’ PD with 
evaluation results 

TL29. During the current school year, have the state and/or district provided 
professional development or other support to the principal and/or other 
leaders of this school on any of the following topics? a. Aligning professional 
development with teacher evaluation results 

TL21. Currently, are principal evaluation results used to develop professional 
development and/or support plans specifically for the principal of your school? 

Principals have the 
opportunity to receive 
financial incentives 
designed to recruit, place, 
and retain staff 

TL26. Currently, do teachers and/or the principal at your school have the 
opportunity to receive any of the following financial incentives? a. 
Signing/recruitment bonuses for beginning to work in this school; b. Retention 
bonuses for continuing to work in the school; c. Performance bonuses; d. 
Increased annual compensation other than bonuses; e. Loan forgiveness; f. 
Tuition reimbursement; g. Housing; h. Financial incentives targeted towards 
increasing the number of staff with English language learner expertise in the 
school 


Source: Surveys of school administrators in spring 2012. 

Note: DA indicates that the question came from the data systems module of the survey. TA 

indicates that the question came from the school turnaround module of the survey. TL 
indicates that the question came from the teachers and leaders module of the survey. 
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Appendix D. Survey Questions Aligned with SIG Practices 


Usage of Practices Promoted by SIG 


Table D.3. Survey Questions Addressing the Learning Time and Community-Oriented Schools 
Topic Area Practices 


Practice 

Survey Questions Addressing the Learning Time and Community-Oriented 

Schools Practice 

Using schedules and 
strategies that provide 
increased learning time 
or increasing the number 
of hours per year that 
school was in session 

TA24. Does your school schedule currently use or offer any of the following? 
a. block scheduling; b. Before- and/or after-school instruction; c. Weekend 
instruction; d. Summer instruction 

TA27. In the current school year, how many hours per day is your school in 
session for students? If the number of hours per day that your school is in 
session varies by day of the week, please record the number of hours per day 
that your school is in session for each day of the week in the box below. 


TA29. In the current school year, how many days per year is your school in 
session for students? 


TA28. In the 2009-2010 school year, how many hours per day was your 
school in session for students? If the number of hours per day that your 
school was in session varied by day of the week, please record the number of 
hours per day that your school was in session for each day of the week in the 
box below. 


TA30. In the 2009-2010 school year, how many days per year was your 
school in session for students? 

Changing policies or 
strategies related to 
parent or community 
engagement 

TA1 2. Since July 2010, did your school implement changes to any of the 
following? i. Policies or strategies related to parent and/or community 
engagement 

State or district provided 
professional 

development on working 
with parents or creating 
a safe school 
environment 

TL29. During the current school year, have the state and/or district provided 
professional development or other support to the principal and/or other 
leaders of this school on any of the following topics? c. Working with parents; 
d. Integrating cultural sensitivity into the school environment 

TL28. Within the past year, has your school engaged in any of the following 
activities? e. Enhanced safety measures in the building; 1. Increased use of 
volunteers (for example, parents) 

Changing discipline 
policies 

TA12. Since July 2010, did your school implement changes to any of the 
following? g. Discipline policies 

Guiding the development 
and implementation of, 
or making changes to, 
nonacademic supports 
or enrichment programs 
for students 

DAI . During the current school year, for which of the following purposes has 
your school used data? c. To guide development and implementation of 
nonacademic supports or enrichment programs (for example, identify how 
many and which students need counseling) 

TA12. Since July 2010, did your school implement changes to any of the 
following? h. Nonacademic supports (for example, mental health supports) for 
students 


Source: Surveys of school administrators in spring 2012. 

Note: DA indicates that the question came from the data systems module of the survey. TA 

indicates that the question came from the school turnaround module of the survey. TL 
indicates that the question came from the teachers and leaders module of the survey. 


D.9 





Appendix D. Survey Questions Aligned with SIG Practices 


Usage of Practices Promoted by SIG 


Table D.4. Survey Questions Addressing the Operational Flexibility and Support Topic Area 
Practices 


Practice 

School has primary 
responsibility for budget, 
hiring, discipline, or 
school year length 
decisions 


State, district, or an 
external support 
provider sponsored by 
the state or district 
provided training or 
technical assistance to 
support school 
improvement efforts or 
that the school received 
support to help 
administrators and 
teachers use data to 
improve instruction 


Survey Questions Addressing the Operational Flexibility and Support Practice 

TA40. Currently, does your school, the district, or the state have primary 
responsibility for decisions in each of the following areas for your school? a. 
Setting student discipline policies; b. Developing the school budget; c. 
Establishing the curriculum (including core texts); d. Setting student 
assessment policies (on assessments other than state-mandated tests); e. 
Staff hiring, discipline, and dismissal; f. Determining the length of the school 
day; g. Determining the length of the school year; h. Setting requirements for 

professional development 

DA8. This school year, has your school received any of the following types of 
support to help school administrators and/or teachers access and use data to 
improve and/or differentiate instruction? For each type of support received, 
please specify the nature of the support that your school received. For 
example, if funding was received, please specify how much funding and the 
purposes for which the funds were used (for example, to buy hardware or 
software, to develop or improve data systems, or to provide training to 
teachers on the analysis and use of data), a. Funds to support school 
investments related to data use; for example, funds to buy hardware or 
software, to develop or improve data systems, or to provide training to 
teachers on the analysis and use of data (Please specify); b. Hardware or 
software to facilitate data use (Please specify); c. Materials on how to access 
and use data to differentiate or improve instruction (Please specify) 

TA39. Since July 2010, have the state and/or district provided any of the 
following types of training or technical assistance to your school? Please 
include assistance provided directly by state or district staff as well as 
assistance funded by the state or district but provided by someone other than 
state or district staff, for example, external consultants or staff from a regional 
office, a. Training or technical assistance on developing and implementing a 
school improvement plan; b. Training or technical assistance on identifying 
curricula, instructional strategies, or school reform models that have been 
shown to be effective at increasing student achievement; c. Training or 
technical assistance on identifying curricula, instructional strategies, or school 
reform models that have been shown to be effective at improving college 
readiness; d. Training or technical assistance on developing strategies to 
recruit and retain more effective teachers 


TA41 . Does your school currently have a state- or district-sponsored external 
support provider(s) or consultant(s) that regularly provides technical 
assistance to your school administrators or instructional staff around school 
improvement efforts? 

Source: Surveys of school administrators in spring 2012. 

Note: DA indicates that the question came from the data systems module of the survey. TA 

indicates that the question came from the school turnaround module of the survey. 
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APPENDIX E 

ADDITIONAL INFORMATION ABOUT ENGLISH LANGUAGE LEARNER-FOCUSED 
ANALYSES FOR THE SIG COMPONENT OF THE EVALUATION 



Appendix E. Additional Information About ELL-Focused Analyses for SIG Usage of Practices Promoted by SIG 


This appendix contains additional infonnation that is directly related to the English language 
learner (ELL)-focused analyses presented in Chapter V. Section A of this appendix lists the 
school administrator survey questions that address the ELL-focused practices aligned with the 
School Improvement Grant (SIG) application criteria (Table E.l). 

Section B of this appendix presents findings from an analysis of the extent to which district 
administrators reported using ELL-focused practices promoted by SIG in spring 2012. These 
findings shed light on the extent to which districts reported providing support to schools for 
ELL-focused practices. We first present the ELL-focused practices aligned with the SIG 
application criteria and the district interview questions that addressed them (Table E.2). For 
readers interested in districts’ reported usage of an individual ELL-focused practice listed in 
Table E.2, we present the extent to which districts reported using the individual ELL-focused 
practices aligned with the SIG application criteria (Figure E.l). We then present a series of 
figures that display the results. We present an analysis of districts’ overall usage of ELL-focused 
practices aligned with the SIG application criteria (Figure E.2) and findings on reported usage of 
ELL-focused practices by districts with above- or below-median ELL populations (Figure E.3) 
and above- or below-median ELL/non-ELL achievement gaps (Figure E.4). 

One important difference between the figures shown in Chapter V of the report and 
section B of this appendix is that the latter have no comparison group. All districts in the study 
sample included schools that were and were not implementing a SIG-fimded intervention model. 
Therefore, in section B, we are not presenting comparisons between districts; rather, we are 
presenting descriptive information about the ELL-focused practices that study districts reported 
using. 

A. School Survey Questions Addressing the ELL-Focused Practices 


Table E.l. School Survey Questions Addressing the ELL-Focused Practices 


ELL-Focused 

Practice 

Survey Questions 

Teachers have the 
opportunity to 
receive financial 
incentives designed 
to increase the 
number of staff with 
ELL expertise 

TL26. Currently, do teachers and/or the principal at your school have the opportunity to 
receive any of the following financial incentives? h. Financial incentives targeted towards 
increasing the number of staff with ELL expertise in the school 

Principals have the 
opportunity to 
receive financial 
incentives designed 
to increase the 
number of staff with 
ELL expertise 

TL26. Currently, do teachers and/or the principal at your school have the opportunity to 
receive any of the following financial incentives? h. Financial incentives targeted towards 
increasing the number of staff with ELL expertise in the school 

Using data on ELLs 
to inform and 
differentiate 
instruction 

DA2. During the current school year, for which of the following purposes has your school used 
data on ELLs? If your school does not have ELLs select- NA. a. To make decisions about 
students’ entry into or exit from ELL status; b. To place ELLs into specialized programs and/or 
classes; c. To track the progress of current ELLs; d. To track the progress of former ELLs; e. 
To inform/improve/differentiate instruction for ELLs 
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Table E.l. (continued) 


ELL-Focused 

Practice Survey Questions 

DA3. Within the past year, did any of the following activities related to data use occur in your 
school? If so, how often did they occur (daily, weekly, monthly, a few times per year, or once 
per year)? For item b below, if your school does not have ELLs, select -NA. b. District staff 
met with you and/or other school staff specifically to review student performance data on ELLs 


TA36. Which of the following strategies/approaches does your school currently use to meet 
the needs of your school’s ELLs? If your school does not have ELLs, select -NA. g. Use data 

on ELLs in school decision making 

Implementing 
strategies, supports, 
or professional 
development to 
meet the needs of 
ELLs 

TA36. Which of the following strategies/approaches does your school currently use to meet 
the needs of your school’s ELLs? If your school does not have ELLs, select -NA. a. Use a 
curriculum that specifically addresses ELLs needs (Please specify); b. Implement instructional 
strategies that specifically address ELLs instruction, or increased progress testing of ELLs 
(Please specify); c. Provide instruction programs specifically designed for ELLs (such English 
as a second language or bilingual programs) (Please specify); d. Provide specialized classes 
for ELLs (such as newcomer class, sheltered content class) (Please specify); f. Provide 
professional development for teacher on providing instruction to ELLs 

DA2. During the current school year, for which of the following purposes has your school used 
data on ELLs? If your school does not have ELLs select- NA. f. To identify professional 
development needs for teachers of ELLs; g. To assess teacher effectiveness with ELLs 

DA3. Within the past year, did any of the following activities related to data use occur in your 
school? If so, how often did they occur (daily, weekly, monthly, a few times per year, or once 
per year)? For item f below, if your school does not have ELLs, select -NA. f. School leaders 
coached teachers on the use of data specifically to improve instruction on ELLs 

TL29. During the current school year, have the state and/or district provided professional 
development or other support to the principal and/or other leaders of this school on any of the 
following topics? e. Ensuring that ELLs acquire the language skills needed to master 
academic content 


TA12. Since July 2010, did your school implement changes to any of the following? For item e 
below, if your school does not have ELLs, select -NA. e. Strategies to meet the needs of ELLs 

TA22. Which of the following topics have been a focus of the professional development 
provided to instructional staff this school year? For item e below, if your school does not have 
ELLs, select -NA. e. Meeting the needs of ELLs 


Providing additional 
services for ELLs 
(such as tutors, 
bilingual aides, or 
an after-school 

program) 

Receiving supports 
from the state or 
local education 
agency to use data 
on ELLs to improve 
or differentiate 
instruction 


TA32. Is your school currently using any of the following methods to organize classes or other 
groups of students for instruction? For item e below, if your school does not have ELLs, select 
-NA. e. Specialized classes for ELLs (such as newcomer class, English as a second 

language, sheltered content). 

TA36. Which of the following strategies/approaches does your school currently use to meet 
the needs of your school’s ELLs? If your school does not have ELLs, select -NA. e. Provide 
additional services for ELLs (such as tutors, bilingual aides, after-school program) (Please 
specify) 


DAI 0. This school year, has your school received any of the following supports to help your 
school access and use data related to ELLs to improve and/or differentiate instruction for 
these students? For each type of support received, please describe the nature of the support 
received. If your school does not have ELLs, select -NA. a. Support to help school staff use 
data to track the performance of ELLs (Please specify); b. Supports to help school staff use 
data to improve or differentiate for ELLs (Please specify) 


Source: Surveys of school administrators in spring 2012. 

ELL = English language learner; NA = not applicable. 
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B. Analysis of Districts’ Reported Usage of ELL-Focused Practices Aligned with 
SIG Application Criteria 

Table E.2. District Interview Questions Addressing the ELL-Focused Practices 


ELL-Focused 

Practice Interview Questions 


Teachers have the 
opportunity to 
receive financial 
incentives designed 
to increase the 
number of staff with 
ELL expertise 


Principals have the 
opportunity to 
receive financial 
incentives designed 
to increase the 
number of staff with 

ELL expertise 

Using data on ELLs 
to inform and 
differentiate 
instruction 


Using data on ELLs 
to implement 
supports or 
professional 
development 


Receiving training 
and technical 
assistance on 
identifying and 
implementing 
strategies to 
address the needs 

of ELLs 

Have designated 
staff or a 

designated office to 
support turnaround 
efforts focused on 
ELLs 


TL29a. Currently, which of the following types of financial incentives are offered by your 
district to teachers working in SIG grantee schools that are implementing one of the four 
intervention models specified by the U.S. Department of Education? h. Financial incentives 
targeted toward increasing the number of staff with ELL expertise in these schools 

TL30. Does your district currently use any of the following other strategies to help recruit 
and retain effective teachers and/or principals in SIG grantee schools implementing one of 
the four intervention models? c. Retention or recruitment efforts targets toward increasing 

the number of staff with ELL expertise in these schools 

TL29b. Currently, which of the following types of financial incentives are offered by your 
district to principals working in SIG grantee schools that are implementing one of the four 
intervention models specified by the U.S. Department of Education? h. Financial incentives 
targeted toward increasing the number of staff with ELL expertise in these schools 


DA6. For which of the following purposes do district staff currently use data specifically on 
ELLs from either the state longitudinal data system or a district data system? a. To make 
decisions about students’ entry into and/or exit from ELL status; b. To place ELLs into 
specialized programs and/or classes; c. To track the progress of current ELLs; d. To track 

the progress of former ELLs; e. To inform/improve/differentiate instruction for ELLs 

DA6. For which of the following purposes do district staff currently use data specifically on 
ELLs from either the state longitudinal data system or a district data system? f. To identify 
professional development needs for teachers of ELLs; g. To assess teacher effectiveness 
with ELLs 

TA26. For which groups does the district provide this additional district-wide support and 

programs? a. ELLs 

TA42. This school year, which of the following types of training and/or technical assistance 
has the state provided to your district to support the improvement efforts of the persistently 
lowest-achieving schools in the district? Please report technical assistance provided directly 
by state staff as well as technical assistance funded by the state but provided by someone 
other than state staff, for example, an external consultant or staff from a regional office, g. 
Training or technical assistance on identifying and implementing strategies to address the 
needs of ELLs 


TA9. Currently, does the district have any of the following organizational or administrative 
structures in place that are specifically intended to support school turnaround efforts 
focused on ELLs? a. District has explicitly designated staff to support school turnaround 
efforts focused on ELLs (but no designated office); b. District has an office explicitly 
designated to support school turnaround efforts focused on ELLs (with designated staff) 


Source: Interviews with district administrators in spring 2012. 

ELL = English language learner. 
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Figure E.l. Districts’ Reported Usage of Individual ELL-Focused Practices Aligned with SIG 
Objectives, Spring 2012 



Extent to 
Which District 
Provides 
Financial 
Incentives to 
Increase Teachers 
with ELL 
Expertise 


Provides Financial 
Incentives to 
Increase Principals 
with ELL 
Expertise 


Extent to 
Which District 
Uses Data on 
ELLs to Inform 
and Differentiate 
Instruction 


Extent to 
Which District 
Uses Data on 
ELLs to Implement 
Supports or PD 


Receives Training 
and TA on 
Implementing 
Strategies to 
Address Needs 
of ELLs 


Extent to Which 
District Has 
Designated Staff 
or Office to 
Support Turnaround 
Efforts Focused 
on ELLs 


ELL-Focused Practice Aligned with SIG Objectives 


| Districts (N=60) 

Source: Interviews with district administrators in spring 2012. 

Note: As described in Chapter II, we selected interview questions that addressed the ELL-focused 

practices aligned with the SIG application selection criteria. The practices shown on the 
horizontal axis of this figure are listed in Table E.2. As described in Chapter II, for each ELL- 
focused practice aligned with the SIG application criteria for which we identified one or more 
interview questions that addressed the practice, we constructed a variable ranging from zero 
to one, with a value of one indicating that the district responded “yes” to all the interview 
questions selected for that practice. The height of each bar represents the mean value of the 
ELL-focused practice (on a scale of zero to one) for each district. For some of the practices 
shown in this figure, multiple interview questions aligned with that practice. In the figure, we 
indicate this using the words “extent to which” at the beginning of the practice, to emphasize 
that the level of usage of that practice is measured using multiple interview questions (as 
opposed to a single, binary measure of whether that practice was used). 

ELL = English language learner; PD = professional development; TA = technical assistance. 
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Figure E.2. Districts’ Reported Usage of ELL-Focused Practices Aligned with SIG Objectives, 
Spring 2012 
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Source: Interviews with district administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table E.2. Each dot in this figure 

represents the number of districts that reported using a particular number of ELL-focused 
practices (out of six examined) that were aligned with the SIG application criteria. The 
number inside each dot is the number of districts represented by the dot; dots that represent 
less than 10 districts have no number inside. For example, 10 districts reported using three of 
the six ELL-focused practices aligned with the SIG application criteria. For three of the ELL- 
focused practices, a “yes” response received one point. In the other three cases, it was 
possible for a school to receive a fraction of one point. See Chapter II for details on the way 
in which the number of ELL-focused practices was determined for each district. The dashed 
line denotes the average number of ELL-focused practices for each district. 
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Figure E.3. Districts’ Reported Usage of ELL-Focused Practices Aligned with SIG Objectives, By 
ELL Population, Spring 2012 



Source: Interviews with district administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table E.2. The figure shows the 

number of ELL-focused practices that districts reported using, by districts that had higher and 
lower ELL populations. Each dot in this figure represents the number of districts that reported 
using a particular number of ELL-focused practices (out of six examined) that were aligned 
with the SIG application criteria. Each dot in this figure represents less than 10 districts, so 
the numbers inside the dots have been removed to protect respondent confidentiality. For 
three of the ELL-focused practices, a “yes” response received one point. In the other three 
cases, it was possible for a school to receive a fraction of one point. See Chapter II for details 
on the way in which the number of ELL-focused practices was determined for each district. 
The dashed line denotes the average number of ELL-focused practices for each district. 
There were no statistically significant differences between districts with higher and lower ELL 
populations at the 0.05 level using a two-tailed test. 
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Figure E.4. Districts’ Reported Usage of ELL-Focused Practices Aligned with SIG Objectives, By 
ELL/Non-ELL Achievement Gap, Spring 2012 
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Source: Interviews with district administrators in spring 2012. 

Note: The practices summarized in this figure are presented in Table E.2. The figure shows the 

number of ELL-focused practices that districts reported using, by districts that had higher and 
lower achievement gaps between ELL and non-ELLs. Each dot in this figure represents the 
number of districts that reported using a particular number of ELL-focused practices (out of 
six examined) that were aligned with the SIG application criteria. Each dot in this figure 
represents less than 10 districts, so the numbers inside the dots have been removed to 
protect respondent confidentiality. For three of the ELL-focused practices, a “yes” response 
received one point. In the other three cases, it was possible for a school to receive a fraction 
of one point. See Chapter II for details on the way in which the number of ELL-focused 
practices was determined for each district. The dashed line denotes the average number of 
ELL-focused practices for each district. There were no statistically significant differences 
between districts with higher and lower ELL/non-ELL achievement gaps at the 0.05 level 
using a two-tailed test. 
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